Study of the Effects of Clinical Decision
Support System’s Incorrect Advice and
Clinical Case Difficulty
on Users’ Decision Making Accuracy
Kamran GOLCHINa, Abdul ROUDSARIa,b,1
a
Centre for Health Informatics, School of Informatics, City University, London, UK
b
School of Health Information Science, University of Victoria, Victoria,
British Columbia, Canada
Abstract. Different Clinical Decision Support Systems (CDSS) are reported to
have different effects on clinicians’ performance and various factors have been
shown to be responsible for that (e.g. system’s advice correctness, case difficulty,
users’ expertise…). The aim of this study is to determine how “advice
correctness” and “case difficulty” affect users accepting/rejecting the comments
of the system and consequently making a right or wrong decision. It was shown
that in difficult cases, users level of making mistakes in clinical decision making
was significantly higher when the comments were wrong. But there was no
statistically significant difference between easy and difficult cases in how users
accepted/rejected correct advice.
Keywords. clinical decision support system, advice correctness, case difficulty,
user performance
Introduction
Clinical Decision Support Systems are "active knowledge systems which use two or
more items of patient data to generate case-specific advice"[1] and support decision
making in clinical processes from diagnosis to treatment, so by definition they play an
important role in reducing medical errors. On the other hand, as physicians are quite
susceptible to following the decision support systems’ comments [2], correct advice is
theoretically helpful to them while the systems’ wrong suggestions are potentially a
risk factor for worsening physicians’ performance. Thus DSS’s effect on physicians’
performance is not always the same and different factors such as system’s advice
correctness, case difficulty and users’ expertise have been shown to affect the final
outcome [3- 8]. This article addresses whether case difficulty and advice correctness
can affect physicians response to DSS, focusing on the following objectives:


1 Abdul Roudsari, School of Health Information Science, University of Victoria, BC, Canada; Email:
abdul@uvic.ca

1. How far are general practitioners (GPs) capable of identifying system’s
incorrect advice (in difficult cases)?
2. How GP’s performances differ between easy and difficult cases (when the
system advice is correct)?
1. Methodology
A clinical decision support system simulator was developed as a web application2, in
which the system’s comments or advice could be manipulated as desired, and in
accordance with the difficulty level of the patient scenarios. Hence, researchers can set
the accuracy of the system by defining the percentage of the comments which will be
wrong (which is not possible in studying real decision support systems). A relational
database was designed and used to store clinical cases in the form of questions, and
each question was related to one correct and some incorrect answers in the form of
advice or system comments. The cases were also divided into three categories
according to their difficulty for general practitioners; easy, intermediate, and difficult.
The DSS simulator published online, and 29 general practitioners participated in
the study. When each GP logged into the application, fifteen randomly selected
questions, five from each category, were presented to him in the form of a quiz with
each question being randomly accompanied by either correct or incorrect advice.
Accuracy of the system was set to 100% for easy case, 80% and 60% for intermediate
and difficult cases respectively. Finally, users were asked whether they agreed with the
advice or not and their answers were recorded by the application.
GPs’ decisions were divided into “Right Decision” (agreeing with correct advice
or rejecting incorrect advice), and “Wrong Decision” (agreeing with incorrect advice
and rejecting correct advice), and this study tried to measure whether the making of
right or wrong decisions is related to the correctness of system comments and/or the
difficulty level of the cases.
2. Results
The first step in the analysis of the found data is data validation and filtering the outlier
data. Pearson Chi square test was used to compare the correctness of the responses
between the participants. The resulting P value was 0.278 with df=28, which is not
statistically significant, and shows that no outlier response is present in this dataset.
To compare the effect of incorrect advice on GPs’ decisions to difficult questions,
the frequency of right/wrong responses (Table 1) were compared using Kendall Tau b
test (both variables were ordinal). This test resulted in P=0.031 which is statistically
significant, and shows that for difficult questions, wrong advice caused more mistakes

2 This CDSS simulator was a dynamic online application developed in .Net framework, and used
MSSQL as the database. This application can be used in other studies by entering clinical scenarios into the
database and easily setting different parameters such as the number of questions shown to each user from
different categories and the proportion of the questions to be randomly accompanied by correct or incorrect
advice. It also retrieves raw data to generate basic reports that can be used in statistical analysis software
such as SPSS.