Fuzzy clustering ensemble for prioritized sampling based on average and rough patterns

Abstract

This paper uses fuzzy clustering to extend a previous prioritized sampling proposal. In many big data problems, modeling an individual object such as a large engineering plant can be a tedious process requiring up to a month of analysis. A solution is to model as many representative objects as possible to represent the entire population. A new object can then use a model (or combination of models) from previously analyzed objects that best matches its characteristics. Since the modeling process can continue indefinitely adding models over time, we prioritize the sampling based on the ability of objects to represent as many characteristics as possible. The approach is demonstrated with a large set of weather stations to create a ranked sample based on hourly and monthly variations of important weather parameters, such as temperature, solar radiation, wind speed, and humidity. The weather patterns are represented using a combination of average and rough patterns to capture the essence of the distribution. The weather stations are grouped using Fuzzy C-Means and the objects with the largest fuzzy memberships are used as the representatives of each cluster. The weather stations representing a combination of different clustering schemes are then ranked based on the number of weather patterns they represent.