An analysis of CAD modeling procedure data collection using synchronous and retrospective think aloud techniques

Abstract

CAD is a critical tool for engineers in the 21st century. To improve CAD usage and education, methods for assessing and evaluating modeling procedures and decision making are necessary. To this end, two common verbal data collection methods are assessed for analyzing CAD modeling procedures. Stimulated recall and concurrent think aloud are compared to each other and screen capture video data. While the concurrent think aloud method seems to increase the necessary modeling time, the think aloud requirement does not affect the proportion of time spent on particular activities. A novel method of using Cohens Kappa with time usage data was implemented to compare the audio methods to screen capture video data. Neither audio method showed significant agreement with the video data when corrected for chance agreement. It is likely that both video and audio data are required to observe significant insights with respect to CAD modeling procedures and decisions. Drawbacks and benefits associated with alternative methods are also highlighted.