An access scheduling tree to achieve optimal freshness in local repositories

Abstract

To achieve improved availability and performance, often, local copies of remote data from autonomous sources are maintained. Examples of such local copies include data warehouses and repositories managed by web search engines. As the size of the local data grows, it is not always feasible to maintain the freshness (up-to-dateness) of the entire data due to resource limitations. Previous contributions to maintaining freshness of local data use a freshness metric as the proportion of fresh documents within the total repository (we denote this as average freshness). As a result, even though updates to more frequently changing data are not captured, the average freshness measure may still be high. In this paper, we argue that, in addition to average freshness, it is important that the freshness metric should also include the proportion of changes captured for each document, which we call object freshness. The latter is particularly important when both the current and historical versions of information sources are queried or mined. We propose an approach by building an access scheduling tree (AST) to precisely schedule access to remote sources that achieves optimal freshness of the local data under limited availability of resources. We show, via experiments, the performance of our approach is significantly higher than a linear priority queue.