Approaching Optimal Duplicate Detection in a Sliding Window

Abstract

Duplicate detection is the problem of identifying whether a given item has previously appeared in a (possibly infinite) stream of data, when only a limited amount of memory is available. Unfortunately the infinite stream setting is ill-posed, and error rates of duplicate detection filters turn out to be heavily constrained: consequently they appear to provide no advantage, asymptotically, over a biased coin toss[8]. In this paper we formalize the sliding window setting introduced by[12, 15], and show that a perfect (zero error) solution can be used up to a maximal window size. Above this threshold we show that some existing duplicate detection filters (designed for the non-windowed setting) perform better that those targeting the windowed problem. Finally, we introduce a “queuing construction” that improves on the performance of some duplicate detection filters in the windowed setting. We also analyse the security of our filters in an adversarial setting.