多数据中心环境中科学大数据应用的数据布局与执行优化研究

项目名称： 多数据中心环境中科学大数据应用的数据布局与执行优化研究

项目编号： No.61502097

项目类型： 青年科学基金项目

立项/批准年度： 2016

项目学科： 自动化技术、计算机技术

项目作者： 张竞慧

作者单位： 东南大学

项目金额： 21万元

中文摘要： 针对多数据中心存储、计算、网络等方面的资源限制引发的科学大数据应用分布式运行效率低下问题，本项目围绕多数据中心环境中科学大数据应用的数据布局和执行优化研究，拟从数据放置、任务指派和作业调度三个方面分别展开优化研究，分别重点解决其中的合理性、时效性、高效性问题。通过感知数据集的协同访问，针对数据中心的存储容量限制，提出访问模式感知的数据放置算法，实现跨数据中心数据访问代价优化；由数据放置驱动，通过权衡多数据中心计算和网络能力，针对数据中心间异构、动态变化的计算资源，提出目标数据驱动的任务指派算法，实现批量任务整体完成时间优化；通过感知作业的初始数据布局以及任务间的数据依赖关系，考虑数据中心的存储和计算资源限制，提出数据布局敏感的作业调度算法，实现跨数据中心执行作业的数据流量优化。本项目将为多数据中心环境下科学大数据应用的高效运行提供切实有效的解决方案，具有重要的理论意义和应用价值。

中文关键词： 多数据中心；数据布局；科学大数据；科学工作流

英文摘要： Small or middle-sized Data Center (DC) in scientific experiments usually has limits in its computing, storage and networking capability. Such limits often cause the inefficient execution of scientific big data application in multi-datacenter environment. This project thus proposes to optimize three key components that impact the overall execution efficiency of scientific big data application: data placement, task assignment and job scheduling. Firstly, a request-pattern aware data placement algorithm is proposed with a joint consideration of the storage limit of each data center. It is able to significantly reduce the data access cost for those request patterns spanning multiple DCs. Secondly, a task assignment algorithm driven by the data placement objective is proposed. This algorithm accounts for the heterogeneity of time-varying computing capability available in each DC. By considering the trade-off between each DC’s computing and inter-DC’s networking capabilities, this algorithm can largely reduce the overall completion time for those batch computing tasks. Thirdly, a job scheduling algorithm that accounts for the storage and computing capability limits of each DC is proposed. This algorithm realizes the optimization of inter-DC data transfer by a joint consideration of the location of job’s input data in multiple DCs and the data dependencies between adjacent tasks of the candidate job. In summary, this project enacts a practical solution for the efficient execution of scientific big data application in multi-DC environment by the optimization of data distribution and job execution.

英文关键词： multi-datacenter ;data distribution;scientific big data;scientific workflow