A Comparative Analysis of Clustering Quality Based on Internal Validation Indices for Dimensionally Reduced Social Media Data

Abstract

Almost all modern industries leverage data analytics to deal with various dimensions of their business like demand forecasting, targeted marketing, and supply chain planning. In addition to historic data, social media data has also become a prominent source of input for data analytics. The key challenges observed with social media data are its huge volume and high dimensions that need to be dealt with. Clustering is the proven strategy in data analytics to segregate the relevant data for processing and thereby reducing the impact of huge volume. Dimensionality corresponds to the diverse features of the data subject being represented. The application of dimensionality reduction techniques can help in reducing the computational intensiveness caused by the curse of dimensionality. This paper covers an experimental analysis using four popular dimensionality reduction techniques – two linear and two nonlinear approaches – to verify the impact of dimensionality reduction on cluster quality using internal clustering validation indices.