Optimization of answer keys for script concordance testing: Should we exclude deviant panelists, deviant responses, or neither?

  • Gagnon R
  • Lubarsky S
  • Lambert C
 et al. 
  • 30


    Mendeley users who have this article in their library.
  • 14


    Citations of this article.


The Script Concordance Test (SCT) uses a panel-based, aggregate scoring method that aims to capture the variability of responses of experienced practitioners to particular clinical situations. The use of this type of scoring method is a key determinant of the tool's discriminatory power, but deviant answers could potentially diminish the reliability of scores by introducing measurement error. (1) to investigate the effects on SCT psychometrics of excluding from the test's scoring key either deviant panelists or deviant answers; (2) to propose a method for excluding either deviant panelists or deviant answers. Using an SCT in radiation oncology, we examined three methods for reducing panel response variability. One method ('outliers') entailed removing from the panel members with very low total scores. Two other methods ('distance-from-mode' and 'judgment-by-experts') excluded widely deviant responses to individual questions from the test's scoring key. We compared the effects of these methods on score reliability, correlations between original and adjusted scores, and between-group effect sizes (panel-residents; panel-students; and residents-students). With a large panel (n = 45), optimization methods have no effect on reliability of scores, correlation and effect size. With a smaller panel (n = 15) no significant effect of optimization methods were observed on reliability and correlation, but significant variation on effect size was observed across samples. Measurement error resulting from deviant panelist responses on SCTs is negligible, provided the panel size is sufficiently large (>15). However, if removal of deviant answers is judged necessary, the distance-from-mode strategy is recommended.

Author-supplied keywords

  • Clinical Reasoning
  • Optimization
  • Panel
  • Reliability
  • Script concordance test

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document


  • Robert Gagnon

  • Stuart Lubarsky

  • Carole Lambert

  • Bernard Charlin

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free