Memory-Adaptive Scheduling for Large Query Execution

Abstract

The execution time of a large query depends mainly on the memory utilization which should avoid disk accesses for intermediate results. Poor memory management can hurt performance and even lead to system thrashing because of paging. However, memory management optimization is hard to incorporate in a query optimizer because of cost estimate errors. In this paper, we address the problem of efficient memory management for large query execution. We propose a static memory allocation scheme applied at start-up time, and a more efficient dynamic execution model which performs memory-adaptive scheduling of the query. Our execution model handles graciously memory overflow and is robust to cost estimate errors. We describe a performance evaluation using a prototype implementation. The experiments with many queries show significant gain (35% to 50% on average and up to 85% in the best case) over static strategies.