ANNALS OF FAMILY MEDICINE ✦ WWW.ANNFAMMED.ORG ✦ VOL. 6, NO. 2 ✦ MARCH/APRIL 2008
154
Data Collection Outcomes Comparing
Paper Forms With PDA Forms
in an Offi ce-Based Patient Survey
ABSTRACT
PURPOSE We compared the completeness of data collection using paper forms
and using electronic forms loaded on handheld computers in an offi ce-based
patient interview survey conducted within the American Academy of Family Phy-
sicians National Research Network.
METHODS We asked 19 medical assistants and nurses in family practices to
administer a survey about pneumococcal immunizations to 60 older adults each,
30 using paper forms and 30 using electronic forms on handheld computers. By
random assignment, the interviewers used either the paper or electronic form
fi rst. Using multilevel analyses adjusted for patient characteristics and clustering
of forms by practice, we analyzed the completeness of the data.
RESULTS A total of 1,003 of the expected 1,140 forms were returned to the data
center. The overall return rate was better for paper forms (537 of 570, 94%)
than for electronic forms (466 of 570, 82%) because of technical diffi culties
experienced with electronic data collection and stolen or lost handheld comput-
ers. Errors of omission on the returned forms, however, were more common
using paper forms. Of the returned forms, only 3% of those gathered electroni-
cally had errors of omission, compared with 35% of those gathered on paper.
Similarly, only 0.04% of total survey items were missing on the electronic forms,
compared with 3.5% of the survey items using paper forms.
CONCLUSIONS Although handheld computers produced more complete data
than the paper method for the returned forms, they were not superior because
of the large amount of missing data due to technical diffi culties with the hand-
held computers or loss or theft. Other hardware solutions, such as tablet comput-
ers or cell phones linked via a wireless network directly to a Web site, may be
better electronic solutions for the future.
Ann Fam Med 2008;6:154-160. DOI: 10.1370/afm.762.
INTRODUCTION
The collection and management of survey data in offi ce-based clini-cal research is challenging. Data collection frequently takes place in association with offi ce visits where it is of secondary importance
to patient care. Data collection may be performed by regular offi ce per-
sonnel who have minimal training in research methods. Once collected,
data must be transmitted to the research offi ce and accurately entered,
coded, and cleaned before analysis. These are time-consuming and exact-
ing tasks that allow opportunities for errors. Electronic data forms may
improve the accuracy and effi ciency of offi ce-based surveys. Electronic
survey forms may be more accurate (or at least more complete) than paper
forms because limits can be imposed on data fi elds, and respondents can
be “forced” to answer each question. If electronic devices are indeed more
accurate and effi cient for offi ce-based surveys, their added expense com-
James M. Galliher, PhD1,2,3
Thomas V. Stewart, BA1
Paramod K. Pathak, PhD4
James J. Werner, PhD5
L. Miriam Dickinson, PhD3
John M. Hickner, MD, MSc6
1AAFP National Research Network,
Leawood, Kansas
2Department of Sociology, University of
Missouri, Kansas City
3Department of Family Medicine, Univer-
sity of Colorado at Denver Health Sciences
Center, Denver
4Department of Epidemiology, Michigan
State University, East Lansing
5 Department of Family Medicine, Case
Western Reserve University, Cleveland, Ohio
6Department of Family Medicine, Univer-
sity of Chicago, Illinois
Confl icts of interest: none reported
CORRESPONDING AUTHOR
James M. Galliher, PhD
National Research Network
American Academy of Family Physicians
11400 Tomahawk Creek Pkwy
Leawood, KS 66211
jgallihe@aafp.org

ANNALS OF FAMILY MEDICINE ✦ WWW.ANNFAMMED.ORG ✦ VOL. 6, NO. 2 ✦ MARCH/APRIL 2008
155
PAPER FORMS VS PDA DATA COLLEC T ION FORMS
pared with paper forms may be justifi ed because of the
lower costs for data management downstream.
Hardware options for electronic data management
in offi ce-based studies include handheld comput-
ers—also referred to as personal digital assistants
(PDAs)—tablet computers, notebook computers, and
desktop computers. Data can be either stored on the
computer for transmission to the research offi ce or
entered directly into a remote database using a Web-
based interface. Pace and Staton1 reviewed the lim-
ited experience of practice-based research networks
(PBRNs) with electronic data collection, describing
potential advantages and challenges of a variety of
approaches. They claimed that “[PDAs] appear to be
the best current option for electronic point-of-care data
collection.” To address this assertion, we studied use of
handheld computers for administering and transmitting
patient survey data. We compared the completeness of
data gathered using paper forms vs handheld comput-
ers in an offi ce-based study in which adults aged 65
years or older were interviewed regarding pneumococ-
cal immunizations. We also describe the technical dif-
fi culties experienced with handheld computers.
METHODS
Recruitment of Study Practices
We sent letters and e-mails in spring 2002 to 164 phy-
sician members of the American Academy of Family
Physicians National Research Network (AAFP NRN)
inviting them to recruit 1 of their nurses or medical
assistants to participate in the Pneumococcal Immu-
nization Study of Older Adults. Forty-four physicians
expressed an interest. Twenty-two did not enroll
because of lack of resources for writing an applica-
tion to obtain local institutional review board (IRB)
approval (75%) or diffi culties with institutional com-
puter security issues such as fi rewalls (25%), leaving 22
practices agreeing to participate.
One nurse or medical assistant from each partici-
pating practice was instructed to interview 60 adult
patients aged 65 years or older about pneumococcal
immunizations—30 with paper forms and 30 with
electronic forms loaded on handheld computers. The
nurses and medical assistants (hereafter referred to as
practice interviewers) were randomly assigned to use
either the paper form or the electronic form fi rst so
that one method or the other would not have an unfair
advantage due to a potential training effect. Data col-
lection began August 22, 2002, and was completed on
September 26, 2003.
One of the authors (T.V.S.) trained each prac-
tice interviewer via individual phone meetings. The
interviewers were instructed to approach consecutive
patients aged 65 years and older who visited the offi ce
for any reason. Follow-up contact between AAFP NRN
staff and the interviewers continued via telephone calls
and e-mails until data collection was completed.
Data Collection Methods and Form Content
The practice interviewers used Sony Clie PEG-T615C
handheld computers (Sony Electronics Inc, San Diego,
California) with the Palm operating system version
4.1 (Palm, Inc, Sunnyvale, Calif). The original data
collection plan called for use of Pendragon Internet
Forms (Pendragon Software Corporation, Libertyville,
Illinois) that would have allowed remote transmission
of encrypted data from each handheld computer to
the Pendragon secure server via the Internet; however,
because institutional fi rewalls protecting many practices
precluded the use of these forms, all but one of the
participating practices mailed the handheld computers
back to the AAFP at the completion of data collection.
Pendragon Forms 3.2 software was used to program the
form into the handheld computers. Electronic data were
downloaded to Microsoft Excel 2000 (Microsoft Corp,
Redmond, Washington) for storage. The practice inter-
viewers mailed the paper forms to the AAFP, where
data were entered by AAFP NRN staff. Staff double-
checked 20% of the paper forms for data entry errors,
which occurred in less than 1% of items.
The 40 survey items were identical on both forms.
The items asked about patients’ knowledge and beliefs
regarding pneumococcal immunizations (7 items); prior
immunization (3 items); reasons for having/not having a
prior immunization (11 items); preference for an immu-
nization at the current offi ce visit and reasons for want-
ing/not wanting an immunization (9 items); selected
demographics (7 items); and administrative issues (3
items). These last questions asked the interviewer to
obtain/supply data on prior pneumococcal immuniza-
tion from the patient’s chart and asked whether the
patient received a pneumococcal immunization at the
current visit and the date of the visit. For the electronic
forms only, all items were “forced choice” (a response
had to be recorded before proceeding) except for the
10 demographic and administrative items.
Outcome Measures
For this study, we counted errors of omission (no
response). We defi ned the survey return rate as the
proportion of completed forms returned relative to
those requested (60 per site). For example, if 50 were
received, this was a survey return rate of 83%. We used
2 additional measures based on returned forms only:
the proportion of returned forms with 1 or more errors
of omission (returned form error rate) and the propor-
tion of returned form items with errors of omission