| 引用本文: | 屈志坚,陈鼎龙,王群峰.配电网监测数据的分布式Map压缩-查询技术[J].电力自动化设备,2017,37(12): |
| QU Zhijian,CHEN Dinglong,WANG Qunfeng.Distributed Map compression-query technology for distribution network monitoring data[J].Electric Power Automation Equipment,2017,37(12): |
|
| 摘要: |
| 针对配电自动化中大量电力监测数据的处理问题，提出了回避聚合操作的配电网监测数据分布式Map压缩-查询新方法。通过将监测数据分布式Map压缩存储，利用HQL查询引擎及压缩接口将分布式Map压缩应用到连接查询的混洗阶段中，减小传递到查询聚合端的数据量，提高压缩数据的查询速度，并推导了时效性的相关公式。以北京某动车段10 kV电力远动监控系统的实测数据为例，搭建了四节点测试集群。压缩导入对比测试表明，分布式Map压缩速度快于分布式Reduce压缩，分布式Map的Map_Deflate压缩处理时间比分布式Reduce_Deflate减少了45.3 %；压缩-查询测试表明，当数据量为2 × 107记录级时，分布式Map的Map_LZO格式压缩-查询耗时大幅降低，比混洗阶段不压缩-查询时减少了31.6 %，验证了分布式Map压缩对加速查询的时效性。 |
| 关键词: 配电网 大数据 集群平台 分布式压缩 压缩-查询 |
| DOI：10.16081/j.issn.1006-6047.2017.12.027 |
| 分类号:TM727;TP311 |
| 基金项目:国家自然科学基金资助项目(51567008)；江西省杰出青年人才计划项目(20162BCB23045)；江西省自然科学基金资助项目(20161BAB206156,20171BAB206044) |
|
| Distributed Map compression-query technology for distribution network monitoring data |
|
QU Zhijian, CHEN Dinglong, WANG Qunfeng
|
|
School of Electrical and Automation Engineering, East China Jiaotong University, Nanchang 330013, China
|
| Abstract: |
| In view of the considerable quantity of monitoring data in distribution automation, an innovative method of distributed Map compression-query for distribution network monitoring data is proposed, in which the aggregation operation can be avoided. The monitoring data is compressed and stored through distributed Map compression. Then, the distributed Map compression is applied to the shuffle stage of join query by HQL query engine and compression interface, which reduces the data quantity transferred to the reducers, and improves the query speed of the compressed data. Besides, the formulas of time effectiveness are deduced. The measured data of a Beijing EMU(Electric Multiple Unit) depot 10 kV power remote monitoring system is taken as an example, where a four-node cluster is constructed to carry out experiments. The compression importing comparison test results demonstrate that the distributed Map compression speed is faster than the distributed Reduce compression, and the Map_Deflate compression processing time decreases by 45.3 % compared to distributed Reduce_Deflate. Meanwhile, the compression-query test results demonstrate that the compression-query time of Map_LZO greatly reduces when the amount of data is 2 × 107 records, which decreases by 31.6 % compared to uncompressed-query in shuffle stage. Hence, these results verify the efficiency of distributed Map compression for accelerating query. |
| Key words: distribution network massive data cluster platform distributed compression compression-query |
