Now You See It, Now You Don't: What Thinking Aloud Tells Us About Clinical Reasoning

Abstract

A ssessing diagnostic reasoning is an important, yet messy endeavor. Imagine you are tasked with rating the reasoning performance of a senior resident while she conducts a history and physical on a patient with a common complaint. What observable behaviors define expert performance? Can listening to the questions she asks, watching the examination maneuvers she performs, or listening to how she describes her conclusions to the patient reliably capture her reasoning abilities? If a different independent observer was watching the same patient encounter, would the 2 observers reliably reach the same conclusions? Would your ratings of this physician's reasoning skills look the same if she were asked to perform the same tasks on a different patient presenting with vague manifestations of a rare disease? As these questions illustrate, diagnostic reasoning is not a discrete, enduring, or reliably measurable skill. Accurate measurement requires an observer to interpret processes that are heavily context dependent, rarely articulated, and often occur below conscious awareness of the observed clinician. 1 Not surprisingly, such inferences can be unreli-able and prone to substantial rater bias. 2 In this issue, Heist and colleagues 3 describe qualitative analyses of discourse obtained from first-year residents who were asked to think aloud while solving clinical vignette-based multiple-choice test items. With a small number of novice subjects and just 6 test items, the study provides a relatively narrow slice of the diagnostic process, yet the findings raise some interesting questions. In practical terms, what purpose could think-aloud protocols in this context serve to advance the field of diagnostic reasoning assessment? It depends on who is asking and how the observations will be used. If the purpose of the think-aloud exercise is to determine test-taking behaviors that lead to correct answers, or what the authors and others have called ''test-wiseness,'' 4 then knowing and sharing such behaviors could help less ''test-wise'' residents to achieve higher test scores, something potentially useful for low performers on high-stakes examinations. This strategy is exploited by test preparation businesses designed, for example, to help premedical students improve their MCAT scores. 5 Insights into test-taking behaviors could help level the playing field among test-takers. If the purpose of the think-aloud exercise is formative assessment, such exercises give residents a chance to explain their reasoning and improve the observer's under-standing of how the resident is prioritizing, sorting, and analyzing the information she is given. In this context, the results could be used on a question-by-question basis to correct learners' misinterpretations, identify knowledge or experience gaps, and formulate strategies for additional learning. A variation on this strategy is used in 1-on-1 clinical teaching encounters 6,7 and during hospital ward rounds. Yet, such coaching for learning is still limited to inferences drawn about residents' reasoning processes that are consciously available to them and verbalized. If the purpose of the think-aloud exercise is to shed light on the relationship between residents' ability to verbalize their reasoning and the accuracy of their diagnoses, previous work analyzing discourse patterns of case presentations may help us understand its value. By analyzing transcripts from oral case presentations, Bordage and Lemieux 8 demonstrated that when clinicians articu-lated their reasoning for problems where they ultimately arrived at the correct diagnosis, these discourse patterns illustrated language structures that signaled a deep and broad understanding of the clinical problem, and these results correlated with other ratings of diagnostic compe-tence. 9 These authors also demonstrated that, with training and calibration, raters could reliably classify the type of discourse residents used when thinking aloud. 9 Thus, with practice and feedback, faculty supervisors could theoreti-cally be trained to listen for discourse characteristics associated with strong reasoning and diagnostic accuracy as well as for the discourse characteristics associated with weak reasoning and diagnostic failure. They could then intervene with targeted interventions to address learners' specific knowledge deficits in a way that could help them access and apply this learning to subsequent clinical experiences.