Bitmap indices for speeding up high-dimensional data analysis

Abstract

Bitmap indices have gained wide acceptance in data warehouse applications and are an efficient access method for querying large amounts of read-only data. The main trend in bitmap index research focuses on typical business applications based on discrete attribute values. However, scientific data that is mostly characterised by non-discrete attributes cannot be queried efficiently by currently supported access methods. In our previous work [13] we introduced a novel bitmap algorithm called Generic Range Eval for efficiently querying scientific data. We evaluated our approach based primarily on uniformly distributed and independent data. In this paper we analyse the behaviour of our bitmap index algorithm against various queries based on different data distributions. We have implemented an improved version of one of the most cited bitmap compression algorithms called Byte Aligned Bitmap Compression and adapted it to our bitmap indices. To prove the efficiency of our access method, we carried out high-dimensional queries against real data taken from two different scientific applications, namely High Energy Physics and Astronomy. The results clearly show that depending on the underlying data distribution and the query access patterns, our proposed bitmap indices can significantly improve the response time of high-dimensional queries when compared to conventional access methods.