基于Hadoop的气象云存储与数据处理平台的研究

项目名称： 基于Hadoop的气象云存储与数据处理平台的研究

项目编号： No.41275116

项目类型： 面上项目

立项/批准年度： 2013

项目学科： 天文学、地球科学

项目作者： 薛胜军

作者单位： 南京信息工程大学

项目金额： 80万元

中文摘要： 为气象行业建立"气象云"是实现海量气象资料共享、智能存储、备份以及数据处理的高效手段。然而，当前云存储架构存在的各种不稳定因素以及气象资料的不同分类、分级方法将会影响云平台的可靠性和管理性能。本项目重点研究基于Hadoop的气象云存储管理与数据处理方法与技术。主要研究内容有：气象云存储平台底层架构中元数据节点单点故障处理方法，数据节点智能管理的解决方法以及融合"异地容灾"机制的数据备份存储方案；结合观测气象资料的种类以及使用级别，研究气象云平台上气象资料分类存储、数据分级备份方法；研究具有"松耦合"特性的"通用计算"类气象作业的任务调度处理模型以及启发式算法求解该类模型的过程。本项目最终将完善当前气象云平台Hadoop集群的底层架构，从而为气象行业建立"专有气象云"服务奠定坚实的理论和技术基础。

中文关键词： 云存储；HA；气象数据；Hadoop；调度

英文摘要： Establish meteorological clouds for meteorological industry is an efficient measure to implement data sharing, intelligent storage, backup and data processing of mass meteorological data. However, there are some problems at present, such as unstable of cloud storage structure, different methods of classifying. Both of them will impact the reliability of the cloud platform and management performance. This project aims at studying the management of meteorological cloud storage and data processing platform based on Hadoop. It contains three parts. The first part needs to find the processing method to handle single point of failure for the metadata that exists in the underlying architecture of the meteorological cloud storage platform, the solution of data nodes' intelligent management and the data's backup and storage solution blending with the mechanism of "disaster-tolerant at distance". The second part tries to find the way that how meteorological data can storage by categories and data backup by classification. The last part attempts to study the processing that how to deal with the loose coupling meteorological jobs of "general computing" by Task Scheduling Model which can be solved by Heuristic Algorithm. The project will finally improve the underlying architecture of Hadoop clusters in current meteorological

英文关键词： Cloud Storage；HA；Meteorological data；Hadoop；Scheduling