Careless responding threatens factorial analytic results and construct validity of personality measure

Abstract

The current research investigates the impact of careless responding on factorial analytic results and construct validity with real data. Results showed that inclusion of careless respondents in data analysis distorts factor loading pattern and hinders recovery of theoretical existing factors. Careless respondents also blur the distinction of theoretically distinct factors, resulting in higher inter-factor correlations. That careless responding may threaten convergent validity also receives limited support. Researchers are advised to exclude careless respondents before statistical analysis. Research on personality relies on collection of accurate data. If a dataset includes a large number of unmotivated respondents, it becomes questionable if conclusion drawn will remain valid. Previous researchers (e.g., Hinkin, 1995) have suggested simple methods to maintain respondents' motivation, such as administering a shorter survey to reduce fatigue or designing questions that are easy to read. Another easy method is to check the existence of careless responding (also called insufficient effort responding; Huang et al., 2012) and exclude the data in analysis (Meade and Craig, 2012; Maniaci and Rogge, 2014; Huang et al., 2015; Kam and Meyer, 2015). However, checking for careless respondents has not been as common as it should be, perhaps because researchers are not fully aware of the effect of careless responding in biasing their research conclusion. It may require more empirical evidence to convince researchers of the importance to control for careless responding. Therefore, in the current research, we will investigate the potential effect of careless responding on factor loading pattern, convergent validity, and discriminant validity. We will use a popular personality measure in our illustration.