On the Evaluation, Management and Improvement of Data Quality in Streaming Time Series

Abstract

The Internet of Things (IoT) technologies plays a key role in the Fourth Industrial Revolution (Industry 4.0). This implies the digitisation of the industry and its services to improve productivity. To obtain the necessary information throughout the different processes, useful data streams are obtained to provide Artificial Intelligence and Big Data algorithms. However, strategic decision-making based on these algorithms may not be successful if they have been developed based on inadequate low-quality data. This research work proposes a set of metrics to measure Data Quality (DQ) in streaming time series, and implements and validates a set of techniques and tools that allow monitoring and improving the quality of the information. These techniques allow the early detection of problems that arise in relation to the quality of the data collected; and, in addition, they provide some mechanisms to solve these problems. Later, as part of the work, a use case related to industrial field is presented, where these techniques and tools have been deployed into a data management, monitoring and data analysis platform. This integration provides additional functionality to the platform, a Decision Support System (DSS) named DQ-REMAIN (Data Quality REport MAnagement and ImprovemeNt), for decision-making regarding the quality of data obtained from streaming time series.