A Group Comparison Test under Uncertain Group Membership

Abstract

An overwhelming majority of articles in psychology compare means, often between multiple groups. However, sometimes we do not know the exact group membership, but only a probability to be in one of the groups. Such information may come from classifiers trained on other datasets, prevalence of group memberships for some parts of the sample, multi-level situations where the group membership is only known as a ratio in an upper level, or expert ratings (e.g., whether a person has a pathological condition or not). We present a simple method that allows to compare group means in the absence of exact knowledge about group membership and investigate the loss of information depending on the probability values theoretically and in a large-scale simulation.