A dynamic data middleware cache for rapidly-growing scientific repositories

1Citations
Citations of this article
7Readers
Mendeley users who have this article in their library.

This article is free to access.

Abstract

Modern scientific repositories are growing rapidly in size. Scientists are increasingly interested in viewing the latest data as part of query results. Current scientific middleware cache systems, however, assume repositories are static. Thus, they cannot answer scientific queries with the latest data. The queries, instead, are routed to the repository until data at the cache is refreshed. In data-intensive scientific disciplines, such as astronomy, indiscriminate query routing or data refreshing often results in runaway network costs. This severely affects the performance and scalability of the repositories and makes poor use of the cache system. We present Delta a dynamic data middleware cache system for rapidly-growing scientific repositories. Delta's key component is a decision framework that adaptively decouples data objects-choosing to keep some data object at the cache, when they are heavily queried, and keeping some data objects at the repository, when they are heavily updated. Our algorithm profiles incoming workload to search for optimal data decoupling that reduces network costs. It leverages formal concepts from the network flow problem, and is robust to evolving scientific workloads. We evaluate the efficacy of Delta, through a prototype implementation, by running query traces collected from a real astronomy survey. © IFIP International Federation for Information Processing 2010.

References Powered by Scopus

Pan-STARRS - A large synoptic survey telescope array

664Citations
N/AReaders
Get full text

Mariposa: A wide-area distributed database system

232Citations
N/AReaders
Get full text

The Story of O2

160Citations
N/AReaders
Get full text

Cited by Powered by Scopus

Sketching the Path to Efficiency: Lightweight Learned Cache Replacement

0Citations
N/AReaders
Get full text

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Cite

CITATION STYLE

APA

Malik, T., Wang, X., Little, P., Chaudhary, A., & Thakar, A. (2010). A dynamic data middleware cache for rapidly-growing scientific repositories. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 6452 LNCS, pp. 64–84). https://doi.org/10.1007/978-3-642-16955-7_4

Readers' Seniority

Tooltip

PhD / Post grad / Masters / Doc 5

83%

Professor / Associate Prof. 1

17%

Readers' Discipline

Tooltip

Computer Science 6

86%

Psychology 1

14%

Save time finding and organizing research with Mendeley

Sign up for free