Use and maintenance of histograms for large scientific database access planning: A case study of a pharmaceutical data repository

Abstract

Scientific databases, and in particular chemical and biological databases, have reached massive sizes in recent years due to the improvement of bench-side high throughput screening tools used by scientists. This rapid increase has caused a shift in the bottleneck in discovery and product development from the bench side to the computational side, thus, creating a need for new computational tools that can facilitate the access and interpretation of such massive data. This paper discusses the design and implementation of the computation of a histogram to speed up access to large pharmaceutical databases. As opposed to traditional histograms in which approximate value distributions is obtained by grouping attribute values into buckets, the computation histogram proposed in this paper records the retrieval time and the calculation time of descriptors in a pharmaceutical drug candidate database. Both on-line and off-line update techniques are proposed to update the computation histogram so that an efficient query plan can be generated. The efficiency of the proposed computation histogram is demonstrated by using a drug candidate database which is used in the pharmaceutical drug discovery process. The histogram allows the result of a query to be either computed using a computational algorithm or retrieved from the database. In addition to the pharmaceutical drug candidate database, the proposed approach is applicable to other scientific databases such as biological and agroscience databases.