Investigating ID3-induced rules from low-dimensional data cleaned by complete case analysis

Abstract

While knowledge discovery in databases techniques require statistically complete data, real world data is incomplete, so it must be preprocessed using completeness approximation methods. The success of these methods is impacted by whether redundancy in large amounts of data overcomes incompleteness mechanisms. We investigate this impact by comparing rule sets induced from complete data with rule sets induced from incomplete data that is preprocessed using complete case analysis. To control the incomplete data construction, we apply the well-defined incompleteness mechanisms missing-at-random and missing-completely-at-random to complete data. Initial results indicate that a medium level of pattern redundancy fails to fully overcome incompleteness mechanisms, and that characterizing an appropriate redundancy threshold is non-trivial. © Springer-Verlag Berlin Heidelberg 2004.