| 引用本文: | 屈志坚,彭翔,王群峰,王汉林.配电网监测数据微批处理的血统链标记容错法[J].电力自动化设备,2019,39(4): |
| QU Zhijian,PENG Xiang,WANG Qunfeng,WANG Hanlin.Lineage chain mark fault-tolerant method for micro-batching of distribution network monitoring data[J].Electric Power Automation Equipment,2019,39(4): |
|
| 摘要: |
| 针对分布式配电自动化系统存在数据量井喷、海量监测数据缺乏高效的分布式故障容错机制的问题，提出一种血统链标记容错新方法。利用弹性分布式数据集、微批计算的记录级容错和血统链标记序列融合处理的设计技巧，实现了分布式数据容错中血统链的追溯和条件标记的自动选择。以铁路配电网监测采集的数据为算例，搭建了4机集群的调度监控平台进行容错测试。以发生频次最高的单数据节点故障为例，测试结果表明：对于包含3×106条监测数据记录的弹性分布式数据集，血统链标记容错模型的集群CPU平均占用率波动小于1.5%，磁盘占用率下降4.2%；当迭代次数达到600、800次时，迭代运算耗时分别降低24.3%和42.9%；所提方法实现500 ms流处理延时的同时，对集群资源的使用情况也具有较好的优化效果，验证了该方法对分布式集群容错的有效性。 |
| 关键词: 配电自动化系统 配电网监测数据 分布式集群 微批计算 血统链标记 流计算 容错 |
| DOI：10.16081/j.issn.1006-6047.2019.04.002 |
| 分类号:TM73 |
| 基金项目:国家自然科学基金资助项目(51567008)；江西省杰出青年人才计划项目(20162BCB23045)；江西省自然科学基金资助项目(20171BAB206044)；江西省科技厅应用培育项目(20181BBE58010) |
|
| Lineage chain mark fault-tolerant method for micro-batching of distribution network monitoring data |
|
QU Zhijian, PENG Xiang, WANG Qunfeng, WANG Hanlin
|
|
School of Electrical and Automation Engineering, East China Jiaotong University, Nanchang 330013, China
|
| Abstract: |
| Aiming at the existing problems of monitoring message blowout and lacking of efficient distributed fault-tolerant mechanism for huge amounts of monitoring data in the distributed distribution automation system, a novel lineage chain mark fault-tolerant method is proposed. The retrospection of lineage chain and the automatic selection of conditional marks in the distributed data fault-tolerant progress are realized by applying the design technique of fusing the resilient distributed data set, the record-level fault-tolerant of micro-batching computing and the lineage chain mark sequence. The monitoring data of railway distribution network are taken as the simulation example and the dispatch monitoring platform of a 4-generator cluster is built to carry out the fault-tolerant test. A single-node fault with the highest frequency is taken as an example, whose test results show that, for the resilient distributed data set including 3×106 monitoring data records, the average CPU utilization rate fluctuation is less than 1.5% and the disk utilization rate decreases by 4.2%;when the number of iteration increases to 600 and 800, the time-consuming decreases by 24.3% and 42.9% respectively; the proposed method not only realizes the stream processing delay of 500 ms, but also has a better optimization effect on the usage of cluster resources, verifying the effectiveness of the method for distributed cluster fault-tolerant. |
| Key words: distribution automation system distribution network monitoring data distributed cluster micro-batching computing lineage chain mark stream computing fault-tolerant |
