Detecting False Identities: A Solution to Improve Web-Based Surveys and Research on Leadership and Health/Well-Being

Abstract

A challenge for leadership and health/well-being research and applications relying on web-based data collection is false identities—cases where participants are not members of the targeted population. To address this challenge, we investigated the effectiveness of a new approach consisting of using internet protocol (IP) address analysis to enhance the validity of web-based research involving constructs relevant in leadership and health/well-being research (e.g., leader–member exchange [LMX], physical [health] symptoms, job satisfaction, workplace stressors, and task performance). Specifically, we used study participants’ IP addresses to gather information on their IP threat scores and internet service providers (ISPs).We then used IP threat scores and ISPs to distinguish between two types of respondents: (a) targeted and (b) nontargeted. Results of an empirical study involving nearly 1,000 participants showed that using information obtained from IP addresses to distinguish targeted from nontargeted participants resulted in data with fewer missed instructed-response items, higher within-person reliability, and a higher completion rate of open-ended questions. Comparing the entire sample against targeted participants showed different mean scores, factor structures, scale reliability estimates, and estimated size of substantive relationships among constructs. Differences in scale reliability and construct mean scores remained even after implementing existing procedures typically used to compare webbased and nonweb-based respondents, providing evidence that our proposed approach offers clear benefits not found in data-cleaning methodologies currently in use. Finally, we offer best-practice recommendations in the form of a decision-making tree for improving the validity of future web-based surveys and research in leadership and health/well-being and other domains.