Big data analytics and knowledge discovery applied to automatic meter readers

Abstract

The volume of data collected by a water utility is constantly growing. In this new era, data are important because guarantees the success of decisions based on the relevant values and underlying information extracted from noisy data. For instance, automatic meter reading (AMR) systems offer households and businesses the chance to understand and reduce their energy and water usage in much greater detail than previously possible, when meter readings were taken once a quarter, or even annually. Moreover, AMR could help utility firms to improve the accuracy of billing and cut visits to properties to read meters. However, with AMR, there is an exponential growth of data: an AMR produces 17500 readings per year, with a single reading every half hour. These data should be first processed in a real-time streaming, in order to be validated before being stored and translated into a metadata model which may be usable in multiple further applications. Thus, utilities have found scaling smart meter management systems difficult to handle. This motivates the use of Big Data technologies in this application domain. On the other hand, applying data analytics and knowledge discovery tools to AMR data combined with other streams of information (data coming from the billing system, call centre service and meteorological information) could help with fraud detection, maintenance requirements prediction, water/energy user consumption patterns determination and response generation to variations in the demand. This chapter presents novel algorithms and methodologies to carry out real-time streaming data processing, data analytics, data quality assessment and improvement, as well as prediction and visualization tasks, at extremely large scale and with diverse structured and unstructured data from multiple sources such as water, power, telecommunication and other utilities, as well as from social media. The algorithms and methodologies will be illustrated using real data coming from several water utilities.