Interviewing AI: Using qualitative methods to explore and capture machines’ characteristics and behaviors

Abstract

In this article, we examine the application of qualitative methods for exploring and capturing the emergent behaviors and characteristics of AI systems. In doing so, we formulate key facets of the ‘interviewing AI’ framework: (1) exploratory familiarization to develop an initial understanding of the AI system's functionalities and responses, (2) systematic investigation through structured probing to elicit behaviors such as hallucinations and manifestations of reasoning, using different prompting approaches, and (3) two complementary approaches - temporal and comparative analyses of AI behavior, examining changes over time or comparing multiple systems at a single point in time. We further discuss (4) potential qualitative analysis methods such as critical discourse analysis or content analysis adapted to theorize and interpret AI behaviors, and (5) triangulation, which integrates qualitative insights from interviewing AI with other methods such as user and expert studies, public interaction records analysis, and quantitative analysis to form a multidimensional and comprehensive understanding of AI systems. Finally, we address (6) ethical considerations by emphasizing transparency, reflexivity, and responsible interpretation of findings to ensure rigorous and contextualized research practices.