“Cake causes herpes?” - promiscuous dichotomisation induces false positives

Abstract

Background: Continuous biomedical data is often dichotomized into two or more groups for analysis, despite long-standing warnings from statisticians that this constitutes bad practice. This dichotomisation is typically discouraged because it reduces statistical power and may obscure important trends. This paper considers another reason to discourage this practice: that dichotomisation is a powerful tool to manipulate data, as dichotomising at an arbitrary yet flexible threshold (which we term ’promiscuous dichotomisation’) represents a powerful researcher degree of freedom. Methods: The motivating question is how probable is it that given a set of uniformly distributed data a threshold can be engineered to produce the illusion of a true effect when none exists? To estimate this, we employed both analytical approaches and Monte-Carlo simulation approaches to quantify the expected number of spurious findings that could arise from manipulating a dichotomous threshold for an arbitrary data set. We also illustrate an example of this with NHANES data, showing how a spurious relationship between blood glucose and herpes status could be engineered. Results: For even a relatively small sample of, a false positive rate of can be observed, rising to over if low counts scenarios are not excluded. With larger samples even with low-count exclusion, false positive rates in excess of for and for are possible, climbing to in excess of and respectively if low-count scenarios were not excluded. For most configurations, manipulation of thresholds was a highly viable methods of crafting a false positive result. Conclusions: It is likely that manipulating cut-off points in measured variables represents a significant source of data manipulation in published science, and the ease of access of larger health databases means this is an issue that is likely to grow in severity. We discuss implications of this, and means of identifying potential promiscuous dichotomisation.