ITM Web Conf.
Volume 20, 2018International Conference on Mathematics (ICM 2018) Recent Advances in Algebra, Numerical Analysis, Applied Analysis and Statistics
|Number of page(s)||16|
|Section||Numerical and Applied Analysis|
|Published online||12 October 2018|
High Performance Energy Prediction using Hadoop with Spark
Ho Chi Minh City University of Technology and Education, 1 Vo Van Ngan, Linh Chieu Ward, Thu Duc District, Ho Chi Minh City, Vietnam
2 Ho Chi Minh City University of Technology, 268 Ly Thuong Kiet, District 10, Ho Chi Minh City, VietNam
3 Tien Giang University, 119 Ap Bac, Ward 5, Tien Giang, My Tho City, Vietnam
Corresponding author: firstname.lastname@example.org
With the increasingly modern development of the power system. Along with that is the data source collected from them is huge. Combined with other systems such as GIS, MDMS-AMR (Automatic Meter Reading), weather forecast and socio-economic indicators. We consider performing an effective analysis of the data sources in order to understand the evolution, characteristics, and modeling of the power consumption system. Thereby predict future energy trends and build bases for the system model. To implement the issues raised, we appreciate using Hadoop platform for storage and segmentation data, enabling better handle large amounts of data initially. Then, the data was analyzed using the scalable machine learning algorithms - MLib was supported and developed on the Spark/SPARKNET platform. The Hadoop framework has recently evolved to the standard framework implementing the MapReduce model. In this paper, we evaluate Hadoop with Mlib/Sparknet performance in both the traditional model of collocated data and compute services as well as consider the impact of separating out the services. Energy modeling from multiple data sources such large may help to understand the change of the system according to consumer demand for practical, predictable trends of energy in the future and provide the basis for building energy models for similar systems.
© The Authors, published by EDP Sciences, 2018
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.
Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.
Initial download of the metrics may take a while.