Evaluating different in-memory cached architectures in regard to time efficiency for big data analysis

Abstract

The era of big data has arrived, and a plethora of methods and tools are being used to manage and analyse the emerging huge volume, velocity, variety, veracity and volatility of information system data sources. In this paper, a particular aspect of a business domain is explored where the primary data being stored/ accessed are not the data value itself (which is highly volatile), but the frequency of its change. Each data frequency has a chain of related data pertaining to it, whose links must be incorporated into this architecture. The volatility of data necessitates the use of in-memory architectures to reduce access/update times. Given these business requirements, different in-memory architectures are examined, using an experiment with sample data, in order to evaluate their worst case response times for a given test set of data analysis/manipulation operations. The results of this experiment are presented and discussed in terms of the most suitable architecture for this type of data, which is in-memory objects linked via hash table links.