Interactive discovery of statistically significant itemsets

Abstract

Frequent Itemset Mining (FIM) is a fundamental data mining task, which consists of finding frequent sets of items in transaction databases. However, traditional FIM algorithms can find lot of spurious patterns. To address this issue, the OPUS-Miner algorithm was proposed to find statistically significant patterns, called productive itemsets. Though, this algorithm is useful, it cannot be used for interactive data mining, that is the user cannot guide the search toward items of interest using queries, and the database is assumed to be static. This paper addresses this issue by proposing a novel approach to process targeted queries to check if some itemsets of interest to the user are non redundant and productive. The approach relies on a novel structure called Query-Tree to efficiently process queries. An experimental evaluation on several datasets of various types shows that thousands of queries are processed per second on a desktop computer, making it suitable for interactive data mining, and that it is up to 22 times faster than a baseline approach.