One of the most important issues in the path to the convergence of HPC and Big Data is caused by the differences in their software stacks. Despite some research efforts, the interoperability between their programming models and languages is still limited. To deal with this problem we introduce a new computing framework called IgnisHPC, whose main objective is to unify the execution of Big Data and HPC workloads in the same framework. IgnisHPC has native support for multi-language applications using JVM and non-JVM-based languages. Since MPI was used as its backbone technology, IgnisHPC takes advantage of many communication models and network architectures. Moreover, MPI applications can be directly executed in a efficient way in the framework. The main consequence is that users could combine in the same multi-language code HPC tasks (using MPI) with Big Data tasks (using MapReduce operations). The experimental evaluation demonstrates the benefits of our proposal in terms of performance and productivity with respect to other frameworks such as Apache Spark. IgnisHPC is publicly available for the Big Data and HPC research community.
翻译:HPC和Big Data汇合过程中最重要的问题之一是软件库的差异。尽管进行了一些研究,但其编程模式和语言之间的互操作性仍然有限。为了解决这个问题,我们引入了名为IgnisHPC的新计算框架,其主要目标是在同一框架内统一执行大数据及HPC的工作量。IgnisHPC利用JVM和非JVM语言对多种语言应用的本地支持。由于MPI被用作其主干技术,IgnisHPC利用了许多通信模型和网络结构。此外,MPI应用程序可以在框架内以有效的方式直接执行,其主要结果是用户可以在同一个多语言代码HPC任务(使用MPI)与大数据任务(使用MapRuce操作)中结合。实验性评估表明我们的建议在诸如Apache Spark等其他框架的绩效和生产率方面的益处。IgnisHPC向大数据与HPC研究界公开提供。