Interrogating the Use of Large Language Models in Qualitative Research Using the Qualifying Qualitative Research Quality Framework

Abstract

Background: Large language models (LLMs) have arrived, recently popularized by the tool ChatGPT, and engineering education researchers have begun experimenting with LLM tools in their work. Considering LLM’s ability to produce competent-sounding text and process data, the use cases seem numerous. As we incorporate these models into our research methodologies, there are concerns that we must address related to research quality. Purpose: Although the concerns with LLMs are well-documented at this point, there is still little work focused on examining the implications of this emerging technology tied to specific quality frameworks to anchor these concerns and, ultimately, develop tangible guiding questions or processes for mitigating the negative consequences associated with this technology. Scope: We discuss the limitations and opportunities of LLMs within the context of the Qualifying Qualitative Research Quality (Q3) Framework (Walther et al. 2013; Walther & Sochacka 2014). Discussion/Conclusions: We provide steps for engineering education researchers curious about experimenting with LLMs in their work. This paper highlights that the issues presented by tools like ChatGPT are by no means new. Still, new methods of reflexivity to actively employ positionality are likely necessary for the ethical adoption of such tools in the field.