A pilot study using Cultural Consensus Analysis to measure Systems-Based Practice performance

Abstract

BACKGROUND: Systems-based practice (SBP) is one of the core competencies for internal medicine training programs. This competency is time consuming and expensive to measure using the suggested 'gold standard' of 360o feedback. Furthermore, there are potential conceptual problems with 360o feedback. Data collection instruments are sponsored by the executive leadership and reflect only their particular goals and values, and they are usually composed of Likert-style questions aimed at evaluating one element at a time. They may not reflect true resource constraint decisions. Cultural Consensus Analysis (CCA) is a standard anthropological technique that determines whether groups hold shared knowledge, and whether they have conflicting preferences. This study aimed to compare an established CCA tool with commonly used 360o feedback instruments. This CCA tool avoids both of the conceptual problems mentioned above, requires only small sample sizes, and is very efficient. METHODS: We performed a cross-sectional study correlating an establishedCCA instrument with the AmericanBoard of Internal Medicine, Patient Satisfaction Questionnaire (ABIM-PSQ) and the American Board of Internal Medicine, Nursing Satisfaction Questionnaire (ABIM-NSQ), commonly used as components of 360o feedback. TheCCA involves sorting 16 cards with statements about things thatmight happen in clinic by order of importance to the subject. The ABIM-PSQ and ABIM-NSQare validated questionnaires.CCA datawas collected by a single research assistant from a quota-based convenience sample of ten residents, ten nursing staff, and ten patients per resident (120 subjects). ABIM-PSQdatawas collected fromall patients andABIM-NSQdata fromall nursing staff simultaneously with the CCA data. The cumulative scores for each resident on the ABIM-PSQ and the ABIM-NSQ were compared to the average difference between patient and resident CCA ranking of two statements: "Doctor and patient agree on goals" (goals) and "Doctor asks what is changing in patient's life (such as a move or major family change)" (changes) using Spearman rank correlation coefficients. These statements were selected because they represent the 'relationship-based care' pole in the previously validated conceptual CCA model of clinic. RESULTS: The correlation between ABIM-PSQ cumulative scores and the difference in patient and resident CCA rankings on 'goals' was - 0.527 (less difference between residents and patients in CCA ranking correlates with higher ABIM-PSQ score). The difference on 'changes' was -0.351. The correlation between ABIM-NSQ cumulative scores and the difference in nursing staff and resident CCA rankings on 'goals' was - 0.086. The difference on 'changes' was -0.415. CONCLUSIONS: SBP is a notoriously difficult competency to evaluate. These moderate correlations between commonly used 360o evaluation instruments (ABIM-PSQ and ABIM-NSQ) and this CCA instrument suggest it can provide an alternative measure of SBP competency. CCA also has conceptual advantages over these other instruments: it is derived from elements that are of concern to all stakeholders in clinic, it forces prioritization of values and it can be presented in a simple graphical format, which makes it an ideal instrument for motivating dialogue about values in teaching clinic and their negotiation.