A STEM dictionary for quantitative text analysis in online education: development and validation

Abstract

Effective online STEM education programs depend on participants’ engagement in discussions about STEM-related topics. The lack of efficient and validated measurement instruments for systematically analyzing such communication, however, limits the collection of behavioral data in this area. This article presents the development and assessment of the main psychometric properties of a LIWC-based STEM dictionary designed to measure STEM-related communication. A series of 10 studies examined the dictionary using classical test theory criteria, including objectivity (particularly objectivity of scoring and interpretation), reliability (parallel-test, split-half, and retest reliability), and validity (construct and criterion validity, including both concurrent and predictive validity). These studies were conducted within the context of a nationwide online mentoring program for girls, focusing on participants’ STEM communication on a secure mentoring platform. Results indicate that the STEM dictionary provides a reliable and valid instrument for measuring STEM-related communication in online educational contexts. We discuss the potential of the dictionary-based approach for evaluating computer-based educational measures in STEM and its use as an efficient method for analyzing communication processes in digital learning environments.