| 引用本文: | 葛磊蛟,王守相,瞿海妮.智能配用电大数据存储架构设计[J].电力自动化设备,2016,36(6): |
| GE Leijiao,WANG Shouxiang,QU Haini.Design of storage framework for big data of SPDU[J].Electric Power Automation Equipment,2016,36(6): |
|
| 摘要: |
| 针对智能配用电数据海量、高维度、多源异构等特点，提出一种大数据存储的三层管理框架设计方案。首先，对智能配用电数据组成进行分类划分。然后，提出智能配用电数据存储的资源层、元数据层和实际数据层的三层管理框架，在资源管理层，应用虚拟化技术、负载均衡和资源调度完成对物理资源的管理；在元数据管理层，使用数据预处理技术对智能配用电的元数据进行分类，采用中间件技术完成XML元数据仓库管理；在实际数据管理层，利用NoSQL技术，通过Map和Reduce的有效结合，完成海量数据的分层分区快速存储。最后，在实验室环境下对该设计方案进行初步的应用，验证了所设计方案能够对结构化、半结构化和非结构化数据进行统一存储，可为智能配用电大数据技术的应用提供基础支撑作用。 |
| 关键词: 大数据 海量数据 框架设计 智能配用电 数据处理 存储 |
| DOI： |
| 分类号: |
| 基金项目:国家高技术研究发展计划（863计划）资助项目（2015AA05203） |
|
| Design of storage framework for big data of SPDU |
|
GE Leijiao1, WANG Shouxiang1, QU Haini2
|
|
1.Key Laboratory of Smart Grid of Ministry of Education，Tianjin University，Tianjin 300072，China;2.State Grid Shanghai Electric Power Company，Shanghai 200122，China
|
| Abstract: |
| A three-layer management framework is designed for the storage of massive，highly dimensional and heterogeneous data of SPDU（Smart Power Distribution and Utilization）. The data components of SPDU are classified and a hierarchical management framework with resource，metadata and actual-data layers is proposed for the storage of SPDU data. The virtualization technology，load balancing and resource scheduling are applied in the resource management layer to realize the management of physical resources. The data preprocessing and middleware technologies are applied in the metadata management layer to respectively classify the metadata of SPDU and manage the warehouse of XML metadata. The NoSQL technology is applied in the actual data management layer to fast store the massive data into different layers and divisions through the effective combination of Map and Reduce. The proposed design scheme is preliminarily applied in the laboratory environment to verify its unified storage of structuralized，semi-structuralized and non-structuralized data，which provides a basic support for the application of big data technology to SPDU. |
| Key words: big data massive data framework design smart power distribution and utilization data processing storage |
