Efficient usage of memory resources in near-real-time data warehousing

Abstract

In the context of near-real-time data warehousing the user's updates generated at data source level need to be stored into warehouse as soon as they occur. Before loading these updates into the warehouse they need to be transformed, often using a join operator between the stream of updates and disk-based master data. In this context a stream-based algorithm called X-HYBRIDJOIN (Extended Hybrid Join) has been proposed earlier, with a favourable asymptotic runtime behavior. However, the absolute performance was not as good as hoped for. In this paper we present results showing that through properly tuning the algorithm, the resulting "Tuned X-HYBRIDJOIN" performs significantly better than that of the previous X-HYBRIDJOIN, and better as other applicable join operators found in literature. We present the tuning approach, based on measurement techniques and a revised cost model. To evaluate the algorithm's performance we conduct an experimental study that shows that the Tuned X-HYBRIDJOIN exhibits the desired performance characteristics. © 2012 Springer-Verlag.