A data quality assessment model and its application to cybersecurity data sources

Abstract

The proliferation of large storage systems such as data lake or big data implies that companies and public institutions need to evaluate the quality of the collected data, in order to ensure that the decisions that take are the most adequate. This fact implies to assess in some cases the own institution, or the data sources that provide the data. Current methods to evaluate data quality are primarily focused on traditional storage systems such as relational databases. In this work, we present a multidimensional data quality evaluation model. We propose a set of data quality dimensions and present an assessment methodology for each of them. The quality of each data source is computed by a mathematical formula that provides us a quality score that let us obtain a ranking of data sources. We also present a software tool that automatically performs the presented quality evaluation model. This tool is flexible enough to be adapted to different datastore systems. In particular, the model is applied over a real datastore of cybersecurity events with data collected from 27 different data sources. They have obtained quality values between −0.125 and 0.719.