Clustering ensemble for prioritized sampling based on average and rough patterns

Abstract

This paper proposes a clustering ensemble for prioritized sampling to tackle a big data problem. The proposal first creates separate clustering schemes of objects using different dimensions of the dataset. These clustering schemes are then combined to create a representative sample based on all the possible combinations of profiles. The resulting clustering ensemble will help system developers to reduce the number of objects that need to be analyzed while making sure that all the profile combinations are comprehensively covered. The proposal further ranks the objects in the sample based on their ability to capture important aspects of each of the criteria. The proposed approach can be used to provide a priority based analysis/modelling over an extended period of time. The prioritized analysis/models will be available for use in a reasonably short period of time. The quality of the analysis/modelling will continuously improve as more and more objects in the sample are processed according to their rank in the sample. The proposal is applied to a large set of weather stations to create a ranked sample based on hourly and monthly variations of important weather parameters, such as temperature, solar radiation, wind speed, and humidity. The experiments also demonstrate how a combination of average and rough patterns help in creating more meaningful profiles.