Is it Possible to Identify Careless Responses with Post-hoc Analysis in EMA Studies?

Abstract

Data quality is a major issue when conducting studies in behavioral sciences. One of the possible threats to data quality in user modeling, in particular in questionnaire studies, is providing careless responses (CR). When responding carelessly, subjects do not pay sufficient attention to the questions and therefore compromise the interpretability of the responses. The aim of the current study was to gain a better understanding of the occurrence and identification of CR in Ecological Momentary Assessment (EMA) studies, where several questionnaires usually are administered daily to the participants over the course of some days, weeks or even months. For this purpose, explorative post-hoc analysis was conducted using the data of an existing EMA study in audiological research. Completion time, variance, skipped items, acquiescence bias and number of textboxes were analyzed as potential indicators for CR both inter- and intraindividually. Furthermore, consistency was examined using linear mixed models and scanning individual questionnaires. Results showed minimal systematic inconsistencies, indicating the absence of large-scale CR. However, this type of analysis might not be appropriate for identifying CR when only occurring occasionally. Moreover, the reliability of indicators of CR might be limited in EMA studies, as the indicators also vary over the course of the study and between different situations. Possibilities for future studies are discussed.