Redesign based on card sorting: How universally applicable are card sort results?

Abstract

Card sort studies can facilitate developers to create an information structure for their website or application. In addition, this human-centered design method provides researchers with insights into the target group’s mental models regarding the information domain under study. In this method, participants sort cards, with excerpts of the website’s or information source’s information on them, into piles or groups. Even though the method lends itself for large numbers of participants, it can be difficult to include sufficient participants in a study to ensure generalizability among large user groups. Especially when the potential user group is heterogeneous, basing the information structure on a limited participant group may not always be valid. In this study, we investigate if card-sort results among one user group (nurses) are comparable to the results of a second (potential) user group (physicians/residents). The results of a formative card sort study that were used to create an antibiotic information application are compared to the results of a second card sort study. This second study was conducted with the aim of redesigning the nurse-aimed information application to meet the (overlapping) needs of physicians. During the first card sort study, 10 nurses participated. In the second card sort study, 8 residents participated. The same set of 43 cards were used in both setups. These cards contain fragments of antibiotic protocols and reference documents that nurses and physicians use to be informed about the use and administration of antibiotics. The participants sorted the cards in individual sessions, into as many categories as they liked. The sorts of both user groups were analyzed separately. Dendrograms and similarity matrices were generated using the Optimal Sort online program. Based on the matrices, clusters were identified by two independent researchers. On these resulting clusters of cards, overlap scores were calculated (between nurse and resident clusters). Differences are compared. The results show that overall, residents reached higher agreement than the nurses. Some overlap between categories is observed in both card sort data matrices. Based on the nurses’ data, more and more specific clusters were created (which in part were observed in the larger residents’ clusters). Based on our findings we conclude that a redesign may not be necessary. Especially when the target group with the lowest prior knowledge levels of the information domain is included in the card sort study, the results can be translated to other groups as well. However, groups with little knowledge will more likely result in lower agreement in the card sorts. Therefore, a larger sample and/or including participants with low and high knowledge of the information domain is advisable.