Please cite
reporting s
ARTICLE IN PRESSIJB-2629; No. of Pages 10
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 0 ) xxx–xxx
journa l homepage: www. int l .e lsev ierhea l th .com/ journa ls / i jmi
A human factors and survey methodology-based design of a
web-based adverse event reporting system for families
Jeremy P a,∗ a chraneb, Roxane Carrc,
Nicola T
a Departmen Britis
British Colum
b Departmen h Co
c Departmen olum
d Department of Family Medicine, University of Alberta, Edmonton, Alberta, Canada
a r t i c
Article histor
Received 19
Received in
24 January 2
Accepted 27
Available on
Keywords:
Adverse eve
Human fact
Patient safe
Patient invo
Reporting sy
1. Int
Recent rep
stay result
cancer, mo
∗ Correspon
V6H 3V4 Ca
E-mail a
1 An adver
rather than
either preve
of a planne
wrong plan
have caused
intercepted
1386-5056/$
doi:10.1016/this article in press as: J.P. Daniels, et al., A human factors and survey methodology-based design of a web-based adverse event
ystem for families, Int. J. Med. Inform. (2010), doi:10.1016/j.ijmedinf.2010.01.016
l e i n f o
y:
December 2009
revised form
010
January 2010
line xxx
nt
ors
ty
lvement
stem
a b s t r a c t
Purpose: Adverse event reporting systems allow healthcare institutions to detect and prevent
recurrence of avoidable patient harm. It is known that standard reporting systems, which
are initiated by clinicians, detect only a minority of chart-documented adverse events. The
objective of the study was to develop a web-based system, the Family Reporting System
(FRS), to elicit adverse event reports from families of children admitted to hospital through
survey methodology and human factors engineering techniques.
Measurements: Face validity and usability were measured via standardized survey instru-
ments. Utility was measured via the rate, typology, degree of harm, likelihood of recurrence,
quality of information, and inter-rater agreement analysis of the reported events.
Results: The FRS has good face validity, excellent usability, and good clinical utility.
Conclusion: The application of survey and human factors methodologies to the design of an
electronic system is an effective means of developing an electronic adverse event reporting
system for the use of families of pediatric patients.
© 2010 Elsevier Ireland Ltd. All rights reserved.
roduction
orts indicate that adverse events1 during hospital
in more deaths per year than deaths from breast
tor vehicle accidents or AIDS [1]. Of these events,
ding author at: BC Children’s Hospital, Department of Anesthesia – Room IL7A, 4480 Oak St., Vancouver, British Columbia,
nada. Tel.: +1 604 719 6772; fax: +1 604 875 3221.
ddress: danielsj@interchange.ubc.ca (J.P. Daniels).
se event is an injury caused by medical management
thepatient’s underlyingdisease. Adverse events canbe
ntable or unpreventable. A Medical Error is the failure
d action to be completed as intended or the use of a
to achieve an aim. A near miss is an error that could
harm but did not reach the patient because it was
[18].
approximately half have been judged preventable [2], thus the
identification of potential or real risks to patients is essential
for improving patient safety.
Currently, most healthcare facilities use adverse event
reporting systems to gather information about safety prob-
lems. These systems require healthcare providers to initiate
– see front matter © 2010 Elsevier Ireland Ltd. All rights reserved.
j.ijmedinf.2010.01.016. Daniels , Ashlee D. King , D. Douglas Co
. Shawd, Joanne Lima, J. Mark Anserminoa
t of Anesthesiology, Pharmacology, and Therapeutics, University of
bia, Canada
t of Neurosurgery, University of British Columbia, Vancouver, Britis
t of Pharmacy, University of British Columbia, Vancouver, British Ch Columbia, Vancouver,
lumbia, Canada
bia, Canada

Please cite ey m
reporting s
ARTICLE IN PRESSIJB-2629; No. of Pages 10
2 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 0 ) xxx–xxx
safety reports. However, many opportunities for improve-
ment are lost because healthcare provider-initiated reports
significantl
Previous re
can provid
healthcare
records [4–
The obj
system, th
event repo
It was hypo
adverse eve
would achi
ity), while p
1.1. Pre
Since To Err
as 100,000 A
[1], a huge i
and reduce
perspective
hospital ad
tries, such
children ex
and loved
an urgent n
events and
tematically
deficiency
providers,
a means to
1.2. Dis
improveme
To increase
tions have
and their f
directly [4–
events mor
Patient rep
ability and
healthcare
investigatio
gate report
In order to
system, wh
cerns, the F
1.3. Res
The aim o
families to
Research ef
goals: (1) re
would be b
rent health
ward, durin
study; (2) fa
of pediatric patients would judge the event categories and lan-
guage to be suitable to the task of reporting adverse events; (3)
ty: fa
enta
be ju
Me
Sel
udy w
n ac
Colu
evan
stud
y stu
ittee
ere r
d gu
t the
clud
Int
roce
view
t or
adve
view
plica
list
uen
iden
3) eq
n ho
and
Int
g sy
mm
Corp
al fo
e we
ystem
terfa
p-do
ed t
her d
tion
pond
g in
ted
wit
tegor
ave
ned
espo
icatithis article in press as: J.P. Daniels, et al., A human factors and surv
y under-represent the true adverse event rate [3].
search has shown that events detected by patients
e valuable learning opportunities not reported by
providers or documented in patients’ medical
14].
ective of the study was to develop a web-based
e Family Reporting System (FRS), to elicit adverse
rts from families of children admitted to hospital.
thesized that the FRS would capture the types of
nts typically experienced at the study hospital and
eve human-centric goals (face validity and usabil-
roviding good utility.
sentation of the problem
is Human, released in 2001, reported that as many
mericans may die each year due to medical errors
nternational effort has been undertaken to identify
harm due to medical care [15]. From the Canadian
, an adverse event occurs in approximately 7.5% of
missions [2], a rate similar to other developed coun-
as the United States where approximately 70,000
perience an adverse event annually [16]. Families
ones suffer by association. Accordingly, there is
eed to provide insight into the nature of adverse
to develop mechanisms by which they can be sys-
reduced on a large scale. Addressing this safety
has become a priority for many researchers and
who have emphasized information technology as
tackle this challenge [17].
covering patient safety opportunities for
nt: giving patients and families a voice
the reporting of adverse events, several institu-
shifted their focus and have given adult patients
amilies the opportunity to report adverse events
14]. These approaches have identifiedmore adverse
e successfully than traditional techniques [9,10].
orts have also been shown to possess good reli-
validity, indicating their potential usefulness in
generally [9,10,12]. However, there are no published
ns of a system designed to allow patient or surro-
ing of adverse events in the pediatric population.
address this gap in patient safety, a web-accessible
ich families can use to report patient safety con-
RS, has been developed.
earch objectives
f this project was to develop a system enabling
routinely identify adverse events and near misses.
forts were divided into the following four separate
levance: the FRS design and structured taxonomy
ased on the adverse events reported to the cur-
care provider reporting system, from the study
g the 1 year prior to the commencement of the
ce validity: clinical experts and parents/guardians
usabili
repres
would
2.
2.1.
The st
pital, a
British
the rel
of the
validit
Comm
FRS w
ents an
ward a
were in
2.2.
The p
ture re
patien
list of
this re
ing ap
initial
subseq
ble inc
care, (
betwee
family
2.3.
learnin
The co
(Datix
hospit
vide th
on a s
The in
of dro
design
to furt
instruc
Res
trainin
consen
sented
six ca
may h
explai
ples. R
by indystem for families, Int. J. Med. Inform. (2010), doi:10.1016/j.ijmedinf.2010.01
ethodology-based design of a web-based adverse event
milies would find the FRS easy to use; (4) utility: a
tive sample of adverse events reported by parents
dged to be clinically useful.
thods
ection of sites and subjects
as conducted at British Columbia Children’s Hos-
ademic tertiary care facility located in Vancouver,
mbia, Canada. Ethics approval was received from
t Institutional Review Boards. The relevance phase
y did not require subject recruitment. For face
dies, members of the hospital’s Family Advisory
, and collaborators who were unfamiliar with the
ecruited. For the usability and utility studies, par-
ardians of childrenadmitted to the general surgical
hospital were recruited. Parents and/or guardians
ed only if they provided written informed consent.
ervention: development of the FRS
ss of developing the FRS began with a litera-
of all English language publications concerning
family-initiated adverse event reporting [18]. A
rse events obtained from the papers produced by
, and the current hospital critical incident report-
tion (Datix Software, London, UK) produced an
of 44 example adverse event items. These were
tly grouped into six main categories of possi-
ts: (1) medication problems, (2) complications of
uipment problems, (4) communication problems
spital staff, (5) communication problems between
staff, and (6) other concerns.
egration into the provider–patient safety
stem
ercial web-based patient safety learning system
oration, London, UK), currently deployed in the
r provider-based reporting, was adapted to pro-
b interface for this study. The FRS design is based
s perspective of adverse events and near misses.
ce is a web page that respondents use via a series
wn lists and pop-up boxes. The web page was
o include introductory information (Fig. 1), links
etails (such as the study information sheet), and
s for use.
ents did not require a log in name, received no
how to use the system, and were recruited and
by a Research Assistant. Respondents are pre-
h a specific question used to identify, from the
ies of errors, the types of adverse events that
occurred during hospitalization. Each category is
using a concise definition and clarifying exam-
ndents must respond to each of the six items
ng if a safety problem had occurred or if it was.016

Please cite
reporting s
ARTICLE IN PRESSIJB-2629; No. of Pages 10
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 0 ) xxx–xxx 3
stopped be
example o
Fig. 2.
Respons
from the d
tions that p
problem (Fi
Table 1 –
Question
Medication
Medication
If any othe
more inf
Was staff a
If you discu
Did you or
If you thinkthis article in press as: J.P. Daniels, et al., A human factors and survey m
ystem for families, Int. J. Med. Inform. (2010), doi:10.1016/j.ijmedinf.2010.01
Fig. 1 – Introductory screen for family-initiated adver
Fig. 2 – Medication problems int
fore occurring (a “near miss” or “close call”). An
f the medication problems category is shown in
es to each question are mandatory. Selecting ‘Yes’
rop-down box provides a list of follow-up ques-
rovide additional information about the identified
g. 3).
Selectin
quick exit
their corres
Responden
Respond
information
patient, an
Design of patient safety problem interface.
problems that occurred:
problems that were stopped before occurring:
r medication problems occurred which were not listed above, or if you woul
ormation on one you observed, please list the details here:
ware of this problem or concern?
ssed this problem or concern with staff, did the discussion meet your need
your family receive an apology from staff?
staff could do anything to prevent this from happening to patients, pleaseethodology-based design of a web-based adverse event
.016
se event reporting system.
erface.
g ‘No’ from the drop-down box allows for a
from that category. The questions asked and
ponding response method is indicated in Table 1.
ts could report problems inmore thanone category.
ents had the option of including identifying
(name, contact information, relationship to the
d willingness to participate in efforts to prevent
Response method
Pop-up box
Pop-up box
d like to provide Free text field
Drop-down list (yes/no)
s? Drop-down list (yes/no)
Drop-down list (yes/no)
select from the list: Pop-up box

Please cite
reporting s
ARTICLE IN PRESSIJB-2629; No. of Pages 10
4 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 0 ) xxx–xxx
F nse
similar occ
tion items.
obtain furt
possibility;
mittee to re
future adve
a similar p
their paren
70% did th
least a uni
greater tha
2.4. Rel
To ensure
provider re
the 1-year p
the Departm
report was
sequent qu
JMA. In the
The numbe
was counte
unsuccessf
2.5. Fac
To establis
experts we
used in the
using www
lect feedba
used in th
componen
anel
eme
ngua
ure
Adv
nal
ons,
Usig. 3 – Additional questions displayed after a positive respo
urrences) following the completion of the six ques-
This was done so that contacting the parent to
her information about a reported event would be a
and for the recruitment of a parent liaison com-
present families during hospital efforts to prevent
rse events. A recent Canadian study performed in
ediatric tertiary care center revealed that most of
t population had internet access (95%), and that
eir banking online. In this population, 42% had at
by a p
improv
and la
proced
Family
additio
questi
2.6.this article in press as: J.P. Daniels, et al., A human factors and survey m
ystem for families, Int. J. Med. Inform. (2010), doi:10.1016/j.ijmedinf.2010.01
versity education, and 63% had a family income
n $50,000 Canadian dollars [19].
evance
that the FRS would be relevant, de-identified
ports of adverse events from the study ward for
eriod, prior to the study period were obtained from
ent of Quality, Safety and Risk Management. Each
matched to one of the six initial questions and sub-
estion items on the FRS by investigators AK and
event of a disagreement, consensus was reached.
r of reports that did not map to an FRS category
d. Revisions were made to the design to include
ully mapped adverse events.
e validity
h the face validity of the FRS, five patient safety
re selected to assess the validity of the wording
FRS (89 items). An online survey application (built
.SurveyMonkey.com, Oregon, USA) was used to col-
ck on the validity of each question or statement
e FRS, according to the Nevo technique [20]. All
ts rated as inadequate or unsuitable were reviewed
To ascertai
who were
recruited fo
safety prob
participant
Usability Q
the FRS th
and/or had
2.7. Uti
To determi
obtained be
parents of
lyzed. Pare
the FRS via
charged fro
classified th
their likelih
of Harm op
• Not a pa
• Near Mis
chance oto one of the six initial question items.
of three of the investigators. Modifications and
nts were then made to the questions statements
ge used in the FRS to improve its validity. This
was repeated with 15 members of the hospital
isory Committee (non-patient safety experts) and
modifications were made to improve validity of
examples and statements.
abilityethodology-based design of a web-based adverse event
.016
n usability of the FRS, parents of children (n=15)
being discharged from the surgical ward, were
r the task of using the FRS to report any patient
lems that had occurred during their stay. After the
s completed this task, the Lewis Computer System
uestionnaire was administered [21]. Any aspect of
at scored below a four on the seven point scale
negative comments was reviewed and revised.
lity
ne the utility of the FRS, all adverse event reports
tween November 1, 2008 and January 31, 2009 from
children hospitalized on the study ward were ana-
nts were consented and allowed to interact with
laptop on the mornings of the day they were dis-
m hospital. Two clinician reviewers reviewed and
e reported events in terms of their degree of harm,
ood of recurrence, and information quality. Degree
tions were:
tient safety issue.
s: Harm almost occurred but was avoided through
r timely action.

Please cite ey methodology-based design of a web-based adverse event
reporting s
ARTICLE IN PRESSIJB-2629; No. of Pages 10
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 0 ) xxx–xxx 5
• Minor Harm: Minor, but temporary, injury occurred to the
patient.
• Moderate Harm: Moderate, but temporary, injury occurred
to the patient.
• Severe Harm: Serious injury, altering hospital stay and/or
requiring additional treatment.
• Death.
Likelihood of recurrence was ranked on a five point scale
between 0 and a 100% (<1%, 1–10%, 11–50%, 51–90%, >90%).
Quality of information was judged on a three point scale (inad-
equate, adequate and excellent). Percent agreement between
reviewers was determined.
2.8. Analysis
The propor
class-taxon
rectionwer
items ranke
on a five p
made to th
FRS. For us
was calcula
of harm, lik
inter-rater
fication wa
3. Re
3.1. Rel
Two hundre
completed
2009). Seven
one FRS cat
more categ
changes to
events invo
distribution
Table 2 – Number of reports mapped to each category.
Category Number of
reports
Percentage
Medication problems 157 55%
Miscommunication
between staff
42 15%
Complication of care 32 11%
Equipment problem 32 11%
Other concerns 16 6%
Miscommunication
between your family
and staff
6 2%
Total 285 100%
3.2. Face validity
3.2.1. Expert face validity review
Experts ran
52 i
, 65%
by
xper
ed th
amb
e eve
orre
d Rea
ngton
uch
iven”
enou
ility)
d no
have
ents
re m
n in
No
s ran
45 it
iew,
by
ers.
Table 3 –
Section ng Items revised after
review
Changes due to expert
comments
Medication 2 3
Complicati 7 0
Equipment 2 2
Miscommu
between
5 1
Miscommu
between
7 4
Other conc 5 4
Follow-up q 2 1
Preventativ 3 2
Total 33 17this article in press as: J.P. Daniels, et al., A human factors and surv
tion of adverse events that did not map to the FRS
omy and the number of revisions required for cor-
e analyzed to gauge relevance. Theproportion of all
d in need of review (having a score of three or less
oint scale), and the number of corrective changes
e FRS were used to determine the face validity of
ability, the average rating in each usability aspect
ted. To estimate utility, the rate, typology, degree
elihood of recurrence, quality of information, and
agreement of the reported events and their classi-
s determined.
sults
evance
d and eighty-five reports paper-based reports were
(from between November 1, 2008 and January 31,
ty-two percent (205) of the reports mapped to only
egory, however, 72 reports (25%) mapped to two or
ories. Nine reports (3%) could not be mapped before
the classification schemeweremade. Eight of these
lved documentation errors. The problem category
is given in Table 2.
Of the
review
flagged
more e
review
reduce
advers
the inc
Kincai
Washi
“too m
drug g
“intrav
readab
line di
which
comm
FRS we
is give
3.2.2.
Parent
Of the
ing rev
flagged
review
Summary of expert reviewers’ validity feedback.
Items considered per
section
Items judged as needi
review
problems 19 6
ons of care 16 11
problems 12 4
nications
staff
10 7
nication
you and staff
11 7
erns 12 11
uestions 2 2
e actions 7 4
89 52ystem for families, Int. J. Med. Inform. (2010), doi:10.1016/j.ijmedinf.2010.01ked 52 items as in need of review (58% of total).
tems ranked by the expert reviewers as needing
were flagged by only one reviewer, and 33% were
two reviewers. No items were flagged by four or
ts. A panel consisting of three of the investigators
e 52 items, and made changes to 33 items (63%) to
iguity and required reading level. An example of an
nt item changed after review is “medicine given in
ct amount” which reads at a Grade 10 level (Flesch-
dability Statistic, Microsoft Word 2007, Redmond,
), which was replaced by the two separate items
drug given” (Grade 1 readability) and “too little
(Grade 4 readability). Another item changed was
s or arterial line did not work correctly” (Grade 11
,whichwas replacedby the two items “intravenous
t work” and “Arterial line did not work”, both of
Grade 5 readability. Additionally, all of the experts’
were reviewed, and an additional 17 changes to the
ade (19% of total items). A summary of the results
Table 3.
n-expert face validity review
ked 45 items as in need of review (58% of total).
ems ranked by the non-expert reviewers as need-
26% were flagged by only one reviewer, 19% were
two reviewers, and 18% were flagged by three
Additionally, one item was flagged by four review-.016

Please cite
reporting s
ARTICLE IN PRESSIJB-2629; No. of Pages 10
6 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 0 ) xxx–xxx
Table 4 – Summary of family member reviewers’ validity feedback.
Section Questionnaire items
considered per section
Questionnaire items
judged as needing
review
Questionnaire items
revised after review
Questionnaire items
revised due to
comments alone
Medication problems 15 14 3 4
Complications of care 14 7 1 1
Equipment problems 9 4 2 2
Miscommunications
between staff
9 4 1 0
Miscommunications
between you and staff
11 4 4 1
Other concerns 8 7 4 0
Follow-up questions 3 1 0 0
Preventative actions 9 4 0 1
Total 78 45 15 9
ers, and the largest number of reviewers flagging a single item
was five, which occurred once. All 45 items were reviewed and
15 changes (33% of total items) were made to reduce ambiguity
and required reading level. The most common reason for not
changing an item when a parent or expert had flagged it as in
need of review was that no explanation was given for how the
item was problematic, and the investigators could not deter-
mine this themselves. In the case where the investigators did
not know the problem with the item, and it was of acceptable
grade level
members’ c
to the FRS
involved re
fashion to
the change
totals in Ta
a list of can
3.3. Us
Twelve per
for further
rective changes to the usability of the FRS. A summary of the
usability results is given in Table 5.
High usability scores (<2.0) were achieved for simplic-
ity, learnability, productivity, and understandability. However,
only moderate usability scores (2.0–2.5) for ease of use, com-
fort, effectiveness, recoverability, and on-screen messages
were obtained, with lower usability scores for on-screen mes-
sages and likeability of using the system.
Uti
ndr
80 c
two
n th
s de
s we
than
port
ed ev
deg
s is g
Table 5 –
Usability
Overall, I a
It is simple
I can effect
I am able t
I am able t
I feel comf
It was easy
I believe I b
The system
Whenever
The inform
It is easy to
The inform
The inform
The organi
The interfa
I like using
This system
Overall, I a, it was not changed. Additionally, all of the family
omments were reviewed and 9 additional changes
(38% of total items) were made. These changes all
ducing the grade level of statements in a similar
the examples given in Section 3.2.1. A summary of
s is given in Table 4. Totals in Table 4 do not match
ble 3 due to deletion of very low scoring items from
didates for FRS category definitions.
ability
cent of the questionnaire items were considered
review, which resulted in a total of nine minor cor-
3.4.
One hu
from 1
thirty-
ipate i
parent
report
more
lem re
report
The
report
Usability experiment results.
statement
m satisfied with how easy it is to use this system.
to use this system.
ively complete my task using this system.this article in press as: J.P. Daniels, et al., A human factors and survey m
ystem for families, Int. J. Med. Inform. (2010), doi:10.1016/j.ijmedinf.2010.01
o complete my task quickly using this system.
o efficiently complete my task using this system.
ortable using this system.
to learn to use this system.
ecame productive quickly using this system.
gives error messages that clearly tell me how to fix problems.
I make a mistake using this system, I recover easily and quickly.
ation (such as on-screen messages) provided with this system is clear.
find the information I need.
ation provided with the system is easy to understand.
ation is effective in helping me complete my task.
zation of information on the system screens is clear.
ce of the system is pleasant.
the interface of this system.
has all the functions and capabilities I expect it to have.
m satisfied with this system.lity
ed and three patient safety problems were reported
onsented discharged patients. Two hundred and
patients were eligible and approached to partic-
e study, resulting in a 78% participation rate (50
clined to participate). These 103 safety problem
re from 73 families as thirty families reported
one patient safety problem. This yields a prob-
ed rate of 40% of discharges. The typology of the
ents is shown in Fig. 4.
ree of harm and likelihood of recurrence of the
iven in Tables 6 and 7.
Average rating out of 7
(1 =max score)
Highest
rating
Lowest
rating
2.2 1 5
1.9 1 5
2.1 1 5ethodology-based design of a web-based adverse event
.016
1.9 1 4
2.1 1 4
2.1 1 6
1.8 1 5
1.7 1 4
2.9 2 4
2.4 1 7
2.2 1 5
2.5 1 5
1.7 1 4
2.0 1 4
2.5 1 5
2.2 1 5
2.8 1 7
2.0 1 4
2.0 1 4

Please cite
reporting s
ARTICLE IN PRESSIJB-2629; No. of Pages 10
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 0 ) xxx–xxx 7
Table 6 – Degree of harm of family reports.
Degree of harm Frequency Number of reports by category
Medication
problems
Complications
of care
Equipment
problems
Miscommunications
between family and
staff
Miscommunications
between staff
Other
Not a patient safety
issue
26% 3 0 3 5 14 10
Near miss 11% 3 1 5 6 10 2
Minor harm 0% 2 4 3 1 1 0
Moderate harm 2% 0 0 0 0 0 0
Severe harm 0% 0 2 0 0 0 0
Death 27% 0 0 0 0 0 0
Cannot evaluate 34% 10 3 3 6 3 3
Table 7 – Likelihood of recurrence of family reports.
Likelihood of
recurrence
Frequency Number of reports by category
Medication
problems
Complications
of care
Equipment
problems
Miscommunications
between family and
Miscommunications
between staff
Other
>90%
51–90%
11–50%
1–10%
<1%
Cannot eva
The qua
adequate o
of harm; bu
quality was
time for jud
of the time
quate for es
of recurren
and gave h
inadequate
hood of rec
was confus
time on rep
Ad
onal
ry ar
Table 8 –
Domain
Medication
Complicati
Equipment
Miscommu
between
Miscommu
between
Other26% 0 4 6
11% 0 0 3
0% 2 0 1
2% 6 1 0
0% 4 0 1
luate 27% 6 5 3
lity of information in the reports was deemed to be
r excellent in 73% of the reports for judging degree
t inadequate 27% of the time. Report information
estimated to be adequate or excellent 80% of the
3.5.
Additi
categothis article in press as: J.P. Daniels, et al., A human factors and survey m
ystem for families, Int. J. Med. Inform. (2010), doi:10.1016/j.ijmedinf.2010.01
ging likelihood of recurrence; but inadequate 20%
. An example of an FRS report judged to be ade-
timating both its degree of harm and its likelihood
ce was “The nurse miscalculated the pain medicine
alf the dose”. An example of an FRS report judged
to estimate both its degree of harm and its likeli-
urrence was “Not sure of dose of medication and
ing”. On average, two reviewers agreed 83% of the
ort classification.
example.
4. Di
4.1. Sig
Several stu
published.
Examples of reports submitted by domain.
errors “Patient developed an itchy rash all over abdomen
ons of care “Child had to have his shunt surgery redone beca
errors “The scale to weigh child did not work. It said he
were in hospital for a few extra days because of it
nications
staff
“Test was repeated because not enough blood wa
vial of blood was required, but the nurses did not
procedure was completed.”
nications
family and staff
“I thought I was sent to my son’s room to wait for
somewhere else. We all finally connected and it tu
would help.”
“When we arrived in urgent care, we provided a lo
constantly repeat ourselves and provide the same
doctors. Is this information not already available?
not good.”staff
13 23 11
2 4 3
0 0 0
0 0 0
0 0 0
3 1 1
ditional examples of FRS reports
examples of reports submitted to the FRS in each
e given in Table 8. The reader may note that theethodology-based design of a web-based adverse event
.016
scussion
nificance
dies of patient reporting adverse events have been
These studies used a wide range of methods
Example
and thighs”
use first one failed”
was losing weight when he was actually gaining. We
!”
s drawn the first time. It ended up that an additional
double-check the amount required until after the first
him. The recovery room nurse thought I’d be waiting
rned out ok. Clearer instructions from less people
t of information to the nurses (4 pages total), but had to
information over and over to the other nurses and
This was complicated by the fact that our English is

Please cite
reporting s
ARTICLE IN PRESSIJB-2629; No. of Pages 10
8 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 0 ) xxx–xxx
and were p
definitive c
method for
has describ
human fac
system int
to report p
on patient
than 25% o
ogy [4,5,23,
the reader
Without de
the reader
tem itself
of human
failed, yet w
in healthca
When so
viewing in-
increase re
experience
usability fo
costly anal
allow for ac
fying adver
options an
classified i
seen with t
event. How
to explain d
satisfaction
questions a
of the FRS.
The app
sistent wit
technology
to underst
adverse eve
ment of Q
similar to a
human-centered design efforts [27], but constitutes more
roblem-domain analysis. The face validity of the FRS
ents
rem
ed u
n, all
o-st
t saf
d ad
s. Th
ed a
g th
impr
nalo
ware
erall
ents’
er fr
ck o
amp
be
e pu
tient
m is
ontin
rega
sen
chers
s pub
prob
y eve
ient
ping
ng le
er an
use o
[32].
FRS
ectin
term
eem
(TheFig. 4 – Typology of FRS reports.
erformed in diverse settings [4–14]. Consequently,
omparisons and conclusions about the optimal
acquiring these reports are not possible. This study
ed the successful use of survey methodology and
tors techniques [22] in developing an electronic
ended to be used by families of pediatric patients
atient safety problems. Reviewing the literature
initiated patient safety reporting shows that less
f published initiatives used web-based technol-
24], and none of them reported system metrics for
to evaluate the quality of their electronic system.
scribing the systems’ face validity and usability,
cannot determine the effect of the electronic sys-
on the outcome measurements. It is this neglect
factors considerations that is often at the root of
ell-intentioned electronic systems put into place
re [25].
liciting adverse event reports, recruiting and inter-
person within hospital and primary care settings
sponse rates [7]. Exclusively asking for personal
s or using open-ended questions may yield higher
of a p
statem
measu
intend
domai
The tw
patien
FRS ha
holder
provid
poratin
many
to be a
in soft
the ov
Par
to diff
feedba
For ex
to not
with th
ing pa
proble
A c
FRS in
no con
resear
article
safety
quentl
for pat
develo
differi
provid
to the
error”
The
of det
that de
were d
time.this article in press as: J.P. Daniels, et al., A human factors and survey m
ystem for families, Int. J. Med. Inform. (2010), doi:10.1016/j.ijmedinf.2010.01
r users, but requires a more time consuming and
ysis. Direct questions and limited response options
curate analysis and provide a structure for classi-
se events and near misses. The limited response
d the fact that a single event could be correctly
nto more than one class is a common difficulty
he classification of any real world complex adverse
ever, closed-ended methods do not allow patients
etails of events, and can inhibit user freedom and
. For these reasons, a combination of closed-ended
nd open-ended narratives was used in the design
roach used in the development of the FRS is con-
h the human-centered design [26] paradigm for
development. The first step in development was
and the user domain by analyzing the types of
nts that had been reported to the hospital Depart-
uality, Risk and Safety in the past year. This is
work-domain analysis, commonly conducted in
also thereb
methods.)
The typ
consistent
communic
portion of
performed
to be a hig
of informa
[34].
Howeve
degree of
tion from
were not s
tion of the
associated
received, h
tinguishingethodology-based design of a web-based adverse event
was also assessed. The function of face validity
ent is to determine how relevant each item is to the
ser. The initial work, understanding the problem-
owed for improved face validity ratings from users.
age approach to evaluating face validity, first with
ety experts and then with parents, ensured the
equate relevance according to both safety stake-
is iterative approach to the face validity testing
n incremental improvement in design. By incor-
ese checkpoints midway through the design cycle,
ovements to the FRS were made. This can be seen
gous to the usability engineering lifecycle [28] used
design, and this approach, in part, is credited for
high usability scores obtained from user testing.
perceptions of adverse events have been shown
om those of healthcare providers [4], and parent
n some of the FRS event options was no exception.
le, several families considered language barriers
a safety problem, a belief that starkly disagrees
blished literature [29]. Future research into defin-
s’ perceptions of what constitutes a patient safety
warranted if their input is to be harnessed.
ually challenging issue during the design of the
rds to patient safety was terminology. There exists
sus on terminology to be used by patient safety
and practitioners [30] and there are fewer than 20
lished on the topic of family reporting of patient
lems [18]. It is still a very nascent field, and conse-
n further away froma consensus-approved lexicon
safety as a whole [30]. Additional challenges when
a standardized patient safety terminology include
vels of healthcare literacy among families [31], and
d risk managers’ fear of malpractice litigation due
f terms which may imply blame, such as “medical
was shown to have good clinical utility. Its rate
g patient safety problems (40%) was higher than
ined by a large chart review study [16], and reports
ed to be of useful quality on average 77% of the
rate of useful reports generated by the FRS was
y higher than that achieved by standard detection
ology of the patient safety problems reported was
with the patient safety literature, where poor
ation and medication errors form a large pro-
presenting adverse events [4,33]. The FRS also
well at identifying near-miss events, well known
hly valuable but very often not reported source
tion in efforts aimed at improving patient safety
r, 27% of the reports could not be evaluated for
harm, mostly due to lack of detailed informa-
the reporter. Thirty-four percent of FRS reports
afety issues at all, indicating a potential limita-
reporting system. These reports were undoubtedly
with dissatisfaction with the healthcare service
owever. Thus, the lay public may have difficulty dis-
between low satisfaction healthcare and unsafe.016

Please cite ey m
reporting s
ARTICLE IN PRESSIJB-2629; No. of Pages 10
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 0 ) xxx–xxx 9
healthcare, which may reflect an inherent limitation of fam-
ily reporting. Dissatisfaction in healthcare however, has been
shown to be a useful predictor of adverse events [35]. Con-
sequently,
as useful,
cific patien
of reports
detail, the
detailed inf
which requ
event when
evaluation.
4.2. Lim
A limitatio
small samp
ing, and re
than a qua
evaluation
agreement
conditions
sures of the
[22]. Percen
instead of
to our data
adverse eve
to perform
ple is a ph
intravenou
scribed am
error (infus
of these cla
expert clas
whether th
tion. Furthe
requires co
unavailable
for such fu
4.3. Fut
Future wo
on a pedia
ilies of pa
complete a
lyzed to d
contrasts w
reports, pa
miss incid
study will
ing familie
increases p
the effect t
relationshi
value of th
mitted from
with collat
records.
5. Conclusion
bina
mple
fam
pati
pme
tha
Whil
e pa
s ar
es n
han
prob
mar
t is
dult
the
dult
use
tion
dver
d fr
t th
rent
an e
rent
pati
he ap
omai
elec
he a
ch a
onic
owle
utho
sista
artic
cript
ers fo
cript
Insti
e n
T. Koh
tionthis article in press as: J.P. Daniels, et al., A human factors and surv
dissatisfaction-based reports may still be regarded
but their value in preventing recurrence of spe-
t safety problems is limited. To reduce the number
which could not be evaluated due to lack of
FRS design could be improved to encourage more
ormation from parents through a forcing function
ires parents include a free text description of the
submitting a report. This would assist in report
itations
n in the development of the FRS was that due to
le sizes, we could not perform hypothesis test-
lied instead on a more qualitative investigation
ntitative one. Additionally, a test–retest reliability
was not performed, and the Pearson coefficient of
between multiple users operating under the same
was not calculated. These are both useful mea-
reliability of a survey instrument such as the FRS
tage agreement was used when measuring utility
a Kappa coefficient or intra-class coefficient due
collection technique. Classification of real world
nts into a small number of categories is impossible
with a high degree of precision. A classic exam-
ysician writes an order for a medication given by
s pump but the dose is given at ten times the pre-
ount—was this a medication error, an equipment
ion pump), or a miscommunication? Because all
ssifications are acceptable, we chose to have one
sify each event, and then asked a second expert
ey would agree with the first expert’s classifica-
rmore, criterion validity was not assessed as this
mparison to a gold standard, which is currently
. It is hoped that the FRS will provide the standard
ture work.
ure work
rk involves an ongoing evaluation of the FRS
tric surgical ward over a 6-month period. Fam-
tients admitted to hospital will be eligible to
report using the FRS. These reports will be ana-
etermine how the pattern of submitted reports
ith a retrospective pattern of provider-initiated
rticularly in terms of adverse event and near-
ence, severity, and problem type. A prospective
be performed to determine if a system offer-
s the opportunity to report patient safety events
rovider reporting. Furthermore, investigation of
he FRS on provider workload and provider–patient
p is planned. Finally, to further investigate the
is approach, the accuracy of the reports sub-
families will be investigated via corroboration
eral information obtained from patients’ medical
A com
were i
allows
atric in
develo
system
ment.
improv
system
care do
researc
safety
Sum
Wha
• A
to
• A
be
tu
• A
an
Wha
• Pa
to
• Pa
to
• T
d
of
• T
su
tr
Ackn
The a
rial as
who p
manus
review
manus
Safety
r e f e r
[1] L.
Naystem for families, Int. J. Med. Inform. (2010), doi:10.1016/j.ijmedinf.2010.01
ethodology-based design of a web-based adverse event
tion of survey and human factors methodologies
mented in the design of a web-based system that
ilies to report adverse events occurring in the pedi-
ent population. The efforts expended during this
nt will translate into increased effectiveness of a
t gives families a voice in patient safety improve-
e enhanced reporting systems on their own cannot
tient safety, well designed and executed reporting
e an essential step towards a world where health-
o harm. In order for this to occur, hospitals need to
d invest in betterways to translate reportedpatient
lems into measurable improvements.
y points
already known on the topic:
patients can report adverse events that occurred
m using a variety of non-validated tools.
patient adverse event reports can be judged to
ful to patient safety analysts at healthcare insti-
s.
se events in healthcare are common, dangerous,
equently preventable.
is study added to our knowledge:
s of pediatric patients will report adverse events
lectronic system.
adverse event reports can be judged to be useful
ent safety analysts at healthcare institutions.
plication of human factors techniques such as
n analysis and usability testing aids the design
tronic adverse event reporting systems.
pplication of survey methodology techniques
s face validity testing aids the design of elec-
adverse event reporting systems.
dgements
rs would like to thank Dina Shafey for edito-
nce, and the families and patient safety experts
ipated in the experimentation reported in this
. Thanks also are extended to two anonymous
r their feedback which improved the quality of the
. Funding was provided by the Canadian Patient
tute and the Canada Institutes for Health Research.
c e s
n, J.M. Corrigan, M.S. Donaldson, To Err is Human,
al Academy Press, Washington, DC, 1999..016

Please cite ey m
reporting s
ARTICLE IN PRESSIJB-2629; No. of Pages 10
10 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 0 ) xxx–xxx
[2] G.R. Baker, P.F. Norton, V. Flintoft, R. Blais, A. Brown, J. Cox, E.
Etchells, et al., The Canadian adverse events study: the
incidence of adverse events among hospital patients in
Canada, CMAJ 170 (2004) 1678–1686.
[3] A.B. Sari, T.A. Sheldon, A. Cracknell, A. Turnbull, Sensitivity
of routine system for reporting patient safety incidents in an
NHS hospital: retrospective patient case note review, BMJ
334 (7584) (2007) 79.
[4] A.J. Kuzel, S.H. Woolf, V.J. Gilchrist, J.D. Engel, T.A. LaVeist, C.
Vincent, et al., Patient-reports of preventable problems and
harms in primary health care, Ann. Med. Fam. 2 (2004)
333–34
[5] T. Agor
undesi
Med. 2
[6] S.N. We
R.B. Da
about a
inciden
[7] L.C. Va
McBrie
of prev
safety
[8] S.M. Ev
percep
(2006) 4
[9] R.L. Ph
Learnin
primar
(2006) 1
[10] R.F. Van
Elder, C
percep
Resear
(2007) 1
[11] H. Nort
McBrie
preven
Health
[12] S.N. We
K.A. Co
in outp
[13] J.H. Wa
interne
Saf. He
[14] D.L.B. S
first re
safety-
(2008) 5
[15] L.L. Lea
what h
2384–2
[16] D. Woo
events
Pediatr
[17] D.W. Ba
inform
(25)) (2
[18] A. King
Anserm
patient reports of adverse events, Qual. Saf. Health Care, in
press.
[19] K. Murto, G.L. Bryson, I. Abushahwan, J. King, D. Moher, K.
El-Emam, et al., Parents are reluctant to use technological
means of communication in pediatric day care, Can. J.
Anaesth. 55 (April (4)) (2008) 214–222.
[20] B. Nevo, Face validity revisited, J. Educ. Meas. 22 (4) (1985)
287–293.
[21] J.R. Lewis, IBM computer usability satisfaction
questionnaires: psychometric evaluation and instructions
for use, Int. J. Hum.-Comput. Interact. 7 (1) (1995) 57–78.
L. Str
actic
organ
Kivla
bens
ent r
mm
N. W
Tess
e We
form
T. Ka
plem
f. He
Zhan
stem
) (200
J. Vic
oduc
lbaum
Danie
amew
chno
Bart
pact
even
n (3)
Thom
ncim
r pat
(1) (2
C. Cu
Ame
terac
M. St
ang,
prob
illwo
T. Kit
utter,
edica
(10)
.L. Pl
e am
tern.
B. Tay
rgma
port
d methis article in press as: J.P. Daniels, et al., A human factors and surv
0.
itsas, P.A. Bovier, T.V. Perneger, Patient reports of
rable events during hospitalisation, J. Gen. Intern.
0 (2005) 922–928.
ingart, O. Pagovich, D.Z. Sands, J.M. Li, M.D. Aronson,
vis, et al., What can hospitalised patients tell us
dverse events? Learning from patient-reported
ts, J. Gen. Intern. Med. 20 (2005) 830–836.
nderheyden, H.C. Northcott, C.E. Adair, C.
n-Morrison, L.M. Meadows, P. Norton, et al., Reports
entable medical errors from the Alberta patient
survey 2004, Healthc. Q. 8 (2005) 107–114.
ans, J.G. Berry, B.J. Smith, A.J. Esterman, Consumer
tions of safety in hospitals, BMC Public Health 6
1–47.
illips, S.M. Dovey, D. Graham, N.C. Elder, J.M. Hickner,
g from different lenses: reports of medical errors in
y care by clinicians, staff and patients, J. Patient Saf. 2
40–146.
Vorst, R. Araya-Guerra, M. Felzien, D. Fernald, N.
. Duclos, et al., Rural community members’
tions of harm from medical mistakes: a High Plains
ch Network (HPRN) study, J. Am. Board Fam. Med. 20
35–143.
hcott, L. Vanderheyden, J. Northcott, C. Adair, C.
n-Morrison, P. Norton, et al., Perceptions of
table medical errors in Alberta Canada, Int. J. Qual.
Care 20 (2008) 115–122.
ingart, J. Price, D. Duncombe, M. Connor, K. Sommesr,
nley, et al., Patient-reported safety and quality of care
atient oncology, J. Qual. Patient Saf. 33 (2007) 83–94.
sson, T.A. MacKenzie, M. Hall, Patients use an
t technology to report when things go wrong, Qual.
alth Care 16 (2007) 213–215.
chwappach, Against the silence: development and
sults of a patient survey to assess experiences of
related events in hospital, BMC Health Serv. Res. 8
9–66.
pe, D.M. Berwick, Five years after To Err is Human:
ave we learned? JAMA 293 (May (19)) (2005)
390.
ds, E. Thomas, J. Holl, S. Altman, T. Brennan, Adverse
and preventable adverse events in children,
ics 115 (January (1)) (2005) 155–160.
tes, A.A. Gawande, Improving safety with
ation technology. Ne analysis, Engl. J. Med. 348 (June
003) 2526–2534.
, J. Daniels, J. Lim, D. Cochrane, A. Taylor, J.M.
ino, Time to listen: a review of methods to solicit
[22] D.
Pr
M
[23] C.
Lo
ev
Co
[24] S.
A.
th
In
[25] B.
im
Sa
[26] J.
sy
(1
[27] K.
Pr
Er
[28] J.
fr
te
[29] G.
Im
pr
(Ju
[30] R.
Ru
fo
21
[31] C.
of
Li
[32] D.
W
im
(M
[33] B.
H
m
34
[34] M
th
In
[35] B.
Be
re
anystem for families, Int. J. Med. Inform. (2010), doi:10.1016/j.ijmedinf.2010.01
ethodology-based design of a web-based adverse event
einer, G.R. Norman, Health Measurement Scales: A
al Guide to Their Development and Use, 2nd ed.,
Kaufmann Publishers Inc., 1993.
han, W. Sangster, K. Nelson, J. Buddenbaum, K.
tein, Developing a comprehensive electronic adverse
eporting system in an academic health center, Jt.
. J. Qual. Improv. 23 (2002) 583–594.
eingart, H.E. Hamrick, S. Tutkus, A. Carbo, D.Z. Sands,
, et al., Medication safety messages for patients via
b portal: the MedCheck intervention, Int. J. Med.
. 77 (2008) 161–168.
rsh, Beyond usability: designing effective technology
entation systems to promote patient safety, Qual.
alth Care 13 (5) (2004) 388–394.
g, Human-centered computing in health information
s. Part 1. Analysis and design, J. Biomed. Inform. 38
5) 1–3.
ente, Cognitive Work Analysis: Toward Safe,
tive and Healthy Computer-Based Work, Lawrence
Associates, Mahwah, NJ, 1999.
ls, S. Fels, A. Kushniruk, J. Lim, J.M. Ansermino, A
ork for evaluating usability of clinical monitoring
logy, J. Clin. Monit. Comput. 21 (5) (2007) 323–330.
lett, R. Blais, R. Tamblyn, R.J. Clermont, B. MacGibbon,
of patient communication problems on the risk of
table adverse events in acute care settings, CMAJ 178
) (2008) 1555–1562.
son, P. Lewalle, H. Sherman, P. Hibbert, W.
an, G. Castro, Towards an international classification
ient safety: a Delphi survey, Int. J. Qual. Health Care
009) 9–17.
tilli, I.M. Bennett, Understanding the health literacy
rica: results of the National Assessment of Adult
y, Orthop. Nurs. 28 (1) (2009) 27–32.
uddert, M.M. Mello, A.A. Gawande, T.A. Brennan, Y.C.
Disclosure of medical injury to patients: an
able risk management strategy, Health Aff.
od) 26 (1) (2007) 215–226.
ch, J.B. Cooper, W.M. Zapol, J.E. Marder, A. Karson, M.
et al., Handoffs causing patient harm: a survey of
l and surgical house staff, J. Comm. J. Qual. Pat. Saf.
(2008) 563–570.
ews-Ogan, M.M. Nadkarni, S. Forren, Patient safety in
bulatory setting. A clinician-based approach, J. Gen.
Med. 19 (July (7)) (2004) 719–725.
lor, E.R. Marcantonio, O. Pagovich, A. Carbo, M.
nn, R.B. Davis, et al., Do medical inpatients who
poor service quality experience more adverse events
dical errors? Med. Care 46 (Feb (2)) (2008) 224–228..016