Biasing effects of non-representative samples of quasi-orders in the assessment of recovery quality of IITA-type item hierarchy mining

Abstract

Inductive Item Tree Analysis (IITA) comprises three data analytic algorithms for deriving reflexive and transitive precedence relations (surmise relations or quasi-orders) among binary items. With the help of simulation studies, the IITA algorithms were already compared concerning their ability to detect the correct precedence relations in observed data. These studies generate a set of surmise relations on an item set, simulate a data set from each of the surmise relations by applying some random response errors, and then try to recover the initial surmise relations from those noisy data. We show that, in the currently published studies however, the representativeness of sampled quasi-orders was not considered or implemented unsatisfactorily. This led to non-representative samples of quasiorders, and hence to biased or wrong conclusions about the quality of the IITA algorithms to reconstruct the underlying surmise relations. In our paper, results of a new, truly representative simulation study are reported, which correct for the problems. On the basis of this study, the three IITA algorithms can now be compared reliably.