Cloud computing has become a major approach to help reproduce computational experiments because it supports on-demand hardware and software resource provisioning. Yet there are still two main difficulties in reproducing big data applications in the cloud. The first is how to automate end-to-end execution of analytics including environment provisioning, analytics pipeline description, pipeline execution and resource termination. The second is an application developed for one cloud, is difficult to be reproduced in another cloud, a.k.a. vendor lock-in problem. To tackle these problems, we leverage serverless computing and containerization techniques for automatic scalable execution and reproducibility, and utilize the adapter design pattern to enable application portability and reproducibility across different clouds. We propose and develop an open-source toolkit that supports 1) fully automated end-to-end execution and reproduction via a single command, 2) automatic data and configuration storage for each execution, 3) flexible client modes based on user preferences, 4) execution history query, and 5) simple reproduction of existing executions in the same environment or a different environment. We did extensive experiments on both AWS and Azure using three big data analytics applications that run on a virtual CPU/GPU cluster. The experiments show our toolkit can achieve good execution performance, scalability and efficient reproducibility for cloud-based big data analytics.
翻译:云计算已成为帮助复制计算实验的主要方法,因为它支持按需提供的硬件和软件资源提供。然而,在复制云中大数据应用方面仍有两个主要困难。第一是如何自动实施终端到终端分析,包括环境提供、分析管道描述、管道执行和资源终止。第二是为一个云开发的应用程序,很难在另一个云中复制,即 a.k.a.供应商锁定问题。为了解决这些问题,我们利用无服务器的计算和集装箱化技术来自动升级执行和再复制,并利用适应设计模式,使应用可移动性和再复制到不同的云层。我们提议并开发一个开放源工具包,支持1)通过单一指令完全自动化终端到终端执行和复制,2)每个执行的自动数据和配置存储,3)基于用户偏好、4执行历史查询和5)简单复制同一环境或不同环境的现有处决。我们用三个大数据系统和Azureer设计模式进行了广泛的实验,使应用能够在不同云层上进行可移动性数据操作的快速测试。