| 摘要: |
| 提出了一种针对海量电能质量数据交换格式文件的快速解析方案。通过快速判断通道序列定义是否变化来避免重复解析相同的数据源记录。利用文件读取线程分解出观测值记录填入任务池中，观测值记录解析线程循环访问任务池，读取观测值记录进行解析后放入数据缓存池，实现了多线程并行解析。基于支持并行扩展的.NET 4.0编程模型实现了所提方案。对现场海量电能质量数据交换格式文件进行解析测试，测试结果显示该方案可显著提高解析效率，并且加速比随CPU核数的增加和数据文件的增多而提高。在四川电能质量一体化数据平台中的成功应用证明了该方案的可靠性和实用性。 |
| 关键词: 电能质量 PQDIF 海量数据 数据处理 线程 并行 解析 .NET 4.0 |
| DOI： |
| 分类号: |
| 基金项目: |
|
| Fast parsing of massive PQDIF files |
|
ZHANG Yi1, YANG Honggeng2
|
|
1.Fujian Electric Power Research Institute，Fuzhou 350007，China;2.School of Electrical Engineering and Information，Sichuan University，Chengdu 610065，China
|
| Abstract: |
| A fast parsing scheme is proposed for PQDIF（Power Quality Data Interchange Format） files，which quickly judges the change of channel series definitions to avoid the repeated parsing of records from same data source. The file reading thread decomposes the observation records and stores them in the task pool while，in parallel，the parsing thread circularly reads and parses the observation records of the task pool and then stores them in the data buffer pool. The proposed scheme is implemented based on the .NET 4.0 programming model which supports the parallel extensions. Test with field massive PQDIF files shows that，the parsing efficiency is significantly improved and the acceleration ratio increases along with the increase of CPU core and data files. Its successful application in Sichuan power quality integrated data platform proves its reliability and practicality. |
| Key words: power quality PQDIF massive data data processing thread parallel programming parsing .NET 4.0 |
