Memory efficient fork-based checkpointing mechanism for in-memory database systems

Abstract

Consistent checkpointing is an essential technique for in-memory databases (IMDBs) to achieve some persistence to data. Towards a fast consistent checkpointing with low overhead and low memory footprint, many consistent checkpointing algorithms have been proposed. However, recent work revealed that the simple fork-based checkpointing scheme used for industrial IMDBs could indeed outperform the state-of-the-arts in terms of average latency, latency spike, and implementation effort. On the other hand, the existing fork-based checkpointing scheme has a memory footprint issue, which remains unsolved; memory usage increases incrementally (up to 2x) during checkpointing for update-intensive workloads. This paper introduces a memory dump based checkpointing scheme, called MDC. With minor operating system supports, our scheme can suppress the increase in memory footprint during checkpointing. By logging the virtual addresses of the objects and by novelly exploiting memory dump, MDC allows pages to be returned to the OS sooner, before the checkpointing process is completely done. This gives the OS the opportunity to reduce the copy-on-write fault overhead because write protection can be ignored when the faulting page is private, thus no page duplication incurs. We implement and apply our scheme into Redis. Extensive evaluations show that our scheme yields a much lower maximum memory footprint and marginally higher throughput in an update-intensive workload scenario.