Quality in Qualitative Evaluation:
A framework for assessing
research evidence
A Quality Framework
Liz Spencer, Jane Ritchie, Jane Lewis and Lucy Dillon
National Centre for Social Research
Government Chief Social Researcher’s Office
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page c3

The report was produced on behalf of the Cabinet Office by Liz Spencer, Jane Ritchie,
Jane Lewis and Lucy Dillon, National Centre for Social Research.
(www.natcen.ac.uk)
The views of this report are the authors’ own and do not necessarily reflect those of the
Cabinet Office.
The full report is entitled: ’Quality in Qualitive Evaluation: A framework for assessing
research evidence’, ISBN: 07715 04465 8.
August 2003
Government Chief Social Researcher’s Office
Crown Copyright 2003
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page c4

CONTENTS
I Introduction 2
II Scope of the Framework 3
III Application of the Framework 6
IV Content of the Framework 7
V Further reading 16
Contents
1
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 1

This document presents a framework for
appraising the quality of qualitative
evaluations. It was developed with particular
reference to evaluations concerned with
the development and implementation of
social policy, programmes and practice.
The framework was devised as part of a
programme of research conducted on
behalf of the Cabinet Office.
The research on which the framework
was built involved:
• a comprehensive review of the literature
on qualitative research methods relating
to standards in qualitative research;
• a review of qualitative research
methods used in Government funded
evaluation studies;
• a review of existing frameworks for
assessing quality in qualitative research;
• exploratory interviews with a range of
people who have an interest in quality
assessment of qualitative research and/or
policy-related evaluations. These included
academics who have written about
qualitative research from either a
theoretical or empirical perspective;
authors of existing frameworks; research
practitioners; commissioners and funders;
and policy-makers who have used
qualitative research evidence in the
development and evaluation of policies;
• a workshop, involving the above groups,
to refine the framework initially developed;
• a trial application of the framework to
a small number of studies.
All these strands of activity have heavily
influenced the content of the framework and
the premises that surround its operation.
A separate report on the study is available.1
The sections below describe the scope,
application and coverage of the framework.
Sections II and III provide important
background information about the premises
underpinning the framework and its intended
usage. It is recommended that these sections
are read by anyone using the framework
for the first time. Those familiar with this
background can move directly to Section IV
where the content of the framework
is presented.
I INTRODUCTION
Qu
al
it
y
in

Qu
al
it
at
iv
e
Ev
al
ua
ti
on
:

A
fr
am
ew
or
k
fo
r
as
se
ss
in
g
re
se
ar
ch

ev
id
en
ce

2
1 Spencer, L., Ritchie, J., Lewis, J. and Dillon, L. (2003). Quality in Qualitative Evaluation: A framework for assessing research evidence,
Government Chief Social Researcher’s Office, London: Cabinet Office.
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 2

The framework has been built on certain
premises surrounding the nature of
qualitative enquiry, how it can be used
for evaluative purposes and how its quality
can be assessed. These premises derive
from extensive debate in the literature
on qualitative research and evaluation.
A brief summary is given here but full
documentation can be found in the
referenced sections of this report.
For the purposes of this framework, the
quality of the qualitative research that
generates the evidence for an evaluation is
seen as lying at the heart of any assessment.
This is because of a primary interest in
evaluations based on empirical enquiry,
which form the majority of government-
based evaluative investigations.
Qualitative research and qualitative
evaluation are therefore seen as broadly
synonymous in terms of the principles
surrounding quality assessment. Because
of this, the framework can also be used
for qualitative research more generally,
irrespective of whether it has an
evaluative purpose.
Qualitative research
Qualitative research aims to provide an
in-depth understanding of people’s
experiences, perspectives and histories in the
context of their personal circumstances or
settings. Among many distinctive features, it
is characterised by a concern with exploring
phenomena from the perspective of those
being studied; with the use of unstructured
methods which are sensitive to the social
context of the study; the capture of data
which are detailed, rich and complex; a
mainly inductive rather than deductive
analytic process; developing explanations
at the level of meaning or micro-social
processes rather than context-free laws;
and answering ‘what is’, ‘how’ and ‘why’
questions. It employs a variety of methods,
including: exploratory interviews; focus
groups; observation; conversation, discourse
and narrative analysis; and documentary
and video analysis.
Qualitative research is used in evaluation
for a range of purposes. For evaluations
of programmes, services or interventions,
these include identifying the factors that
contribute to successful or unsuccessful
delivery; identifying outcomes (intended
or unintended) and how they occur;
examining the nature of requirements of
different groups within the target population;
exploring the contexts in which policies
operate; and exploring organisational aspects
of delivery. Qualitative research can also
be used in advance of policy development
or implementation, for example, to examine
an issue or problem that is poorly understood
or to inform the kind of intervention required.
A further use is to help develop evaluative
criteria where these are unclear or where
alternative criteria are sought.
II SCOPE OF THE FRAMEWORK
ii

Scope
of
the
framework
3
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 3

Quality standards in qualitative research
Despite many different approaches and
traditions within qualitative and evaluative
research, there are widespread concerns
about quality. There is also shared interest
in issues such as ‘rigour’; the need for
principles of practice to be made manifest;
the importance of sound or ‘robust’
qualitative research evidence; and in the
relevance and utility of research.
The nature of quality assessment
in qualitative research
There is debate in the literature about
whether the concepts of quality used to
assess qualitative research should be roughly
the same as, parallel to, or quite different
from those used to assess quantitative
research. This framework is based on the
view that the concerns which lie behind
customary conceptions of quality have
relevance for qualitative enquiry but need
to be reformulated – and assessed quite
differently – within the domain of qualitative
research. In other words, qualitative research
should be assessed on its ‘own terms’ within
premises that are central to its purpose,
nature and conduct.
The formalisation of quality standards
There is some debate about the extent to
which quality assessment of qualitative
inquiry can be formalised. Alongside this
there have been increasing calls for guidance
about quality assessment so that criteria
appropriate to qualitative research (rather
than those imported from other research
traditions) are used. This, in turn, has led
to the generation of a number of checklists,
guidelines and lists of appraisal questions
for assessing qualitative research.
This framework draws heavily on previously
developed quality criteria, both from the
general methodological literature and from
pre-existing guidelines. However, it also
takes heed of the persistent concern that
formalised criteria should avoid being
rigidly procedural or over-prescriptive.
It has therefore been devised to aid informed
judgement, not mechanistic rule-following.
Because there has been an opportunity to
build on other frameworks, it is
comparatively comprehensive in its coverage.
It is also distinctive in making explicit the
philosophical assumptions within which it
operates; and in the level of specificity of
the quality ‘indicators’ on which to base
an assessment.
Philosophical assumptions
There are numerous approaches, paradigms,
schools and movements encompassed within
what is broadly termed ‘qualitative research’.
They vary in terms of the ontological,
epistemological and methodological
assumptions on which they are based.
The same issues arise in relation to different
types of evaluation. A proliferation of
approaches to evaluation exist, which,
again, differ in terms of their philosophical,
ideological and methodological premises.
In both cases, the various traditions are often
categorised under labelled movements or
schools although their identities are not
always consistently described.
Qu
al
it
y
in

Qu
al
it
at
iv
e
Ev
al
ua
ti
on
:

A
fr
am
ew
or
k
fo
r
as
se
ss
in
g
re
se
ar
ch

ev
id
en
ce

4
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 4

It was not felt possible to design a single set
of quality markers that would be appropriate
for all the different qualitative and evaluative
traditions that exist. The assumptions
on which they are based are too diverse
– in some cases contradictory – and the
conception of what matters in terms of
‘quality’ can vary with these assumptions.
Instead, an ‘elemental’ approach has been
used in which a range of premises about
the nature of social reality and how the
social world can be investigated have been
identified and those which underpin the
framework specified. It is thought that these
assumptions are broadly those within which
most government-funded qualitative inquiry
operates. Taking this approach means that
the framework will be relevant for a range
of types of qualitative evaluations including
practice evaluation, policy development and
appraisal as well as evaluations of particular
interventions, schemes or programmes.
It will be the assumptions within which an
evaluation operates, rather than the model
of qualitative research or evaluation used,
that will determine whether the framework
will be of value to assess its quality.
Choice of appraisal items
There are numerous appraisal items that could
have been included in the framework.
Those selected were chosen on the basis that:
• they operate within the boundaries of
the philosophical assumptions prescribed;
• they offer a series of readings on core
principles concerning the contribution,
defensibility, rigour and credibility of
a qualitative study;
• they cover different stages and processes
within qualitative enquiry;
• they are recurrently cited as markers
of quality in the literature, in pre-existing
frameworks and in the interviews
conducted for this study.
ii

Scope
of
the
framework
5
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 5

The framework has been designed primarily
to assess the outputs of qualitative inquiry –
that is reports, papers and journal articles.
This has inevitably affected both the items
covered and, most crucially, how they can
be demonstrated for assessment. There is,
for example, heavy emphasis on the quality
of the evidence and the analysis that has
informed it and rather less on aspects of
conduct that cannot be well judged from
written output.
Some translation of the criteria included can
be made to assess proposals. Guidance notes
on this are given in Section IV. It is also
hoped that the framework will have a wider
educational function in the preparation of
research protocols, the conduct and
management of research and evaluation
and in the training of social researchers,
both within and outside government.
Relevance for different
qualitative methods
The framework has been designed with a
particular focus on the methods used most
extensively in government-based evaluations,
namely, interviews, focus groups, observation
and documentary analysis. Nevertheless, the
principles that have guided the design of
the framework, and many of the questions
suggested for appraisal, will have application
to a wider range of qualitative methods
(e.g. linguistic analysis, historical and archival
analysis, multimedia methods etc.). However,
quality indicators that are particularly relevant
to other methods will need to be added.
Use of multiple methods
It is common to find that evaluations use
a mix of methods to collect the required
information. Sometimes these are a
combination of qualitative methods but
in other cases a mix of qualitative and
quantitative approaches. Although some
of the appraisal questions in the framework
are method generic (e.g. Qs 1–3, 5, 6 etc.),
others are specific to qualitative methods
(e.g. Qs 11–13). It is therefore recommended
that to assess any quantitative components
of an evaluation, quality criteria that are
specific to the method concerned will need
to be added.
Application to different policy domains
The framework has been designed to assess
qualitative evaluation across the spectrum
of substantive fields covered by Government
Departments. However, it could well be that
assessment within a particular substantive
area might need to address some additional
questions. If so, these can be added to the
framework as necessary.
III APPLICATION OF THE FRAMEWORK
Qu
al
it
y
in

Qu
al
it
at
iv
e
Ev
al
ua
ti
on
:

A
fr
am
ew
or
k
fo
r
as
se
ss
in
g
re
se
ar
ch

ev
id
en
ce

6
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 6

The framework is built on a set of principles
around which to frame and structure
appraisal questions that might be asked
of a piece of work in order to critically assess
its quality. In each case, a set of quality
indicators is listed – features that will help
to form a judgement about how well the
appraisal question has been addressed.
Further questions might also be added
depending on the purpose of the research
and the approach it uses. It is then for the
assessor to judge overall merit, based on
the questions and indicators that are most
relevant to the evaluation concerned.
Guiding principles
There are four2 central principles that
underpin the content of the framework.
All of these are based on themes that are
highly recurrent in the literature and in
the interviews conducted for the study.
They advise that research should be:
• contributory in advancing wider
knowledge or understanding about
policy, practice, theory or a particular
substantive field;
• defensible in design by providing
a research strategy that can address
the evaluative questions posed;
• rigourous in conduct through the
systematic and transparent collection,
analysis and interpretation of
qualitative data;
• credible in claim through offering
well-founded and plausible arguments
about the significance of the
evidence generated.
These principles are presented at a
sufficiently high level of abstraction that
they would apply to a diversity of qualitative
approaches. Indeed, most of them are
simply emblems of sound and logical
enquiry, whatever its form or purpose.
Appraisal questions
The guiding principles have been used to
identify 18 appraisal questions to aid an
assessment. The questions are listed in
column a) of the framework. Between them,
they cover all of the key features and
processes involved in qualitative enquiry.
They begin with assessment of the findings,
move through different stages of the
research process (design, sampling,
data collection, analysis and reporting)
and end with some general features of
research conduct (reflexivity and
neutrality, ethics and auditability).
It is suggested that the findings of the
enquiry are given attention first, even though
this is not a logical procedural order. This is
because the nature of the evidence presented
will help in assessing features of the research
process (for example, the quality of the
data collected, the visibility and logic of the
analytic process). However, if readers prefer
IV CONTENT OF THE FRAMEWORK
IV

Content
of
the
framework
7
2 Two of the four principles deal with the way in which a study has been devised and conducted. This is not surprising since,
in qualitative research, there are no ‘validated’ instruments or standardised methods, which means that quality cannot be
assumed on the basis that certain methods have been used, but must be demonstrated in each case.
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 7

to look at research design and conduct
before considering the evidence, they
will need to return to Qs 6–18 before
completing the assessment.
Quality indicators
Beside each question, there is a series of
quality indicators (column b) which will
help in answering the appraisal question.
These provide pointers to the kinds of
information needed to judge whether
or not the quality feature concerned has
been secured. They are not intended to
be comprehensive and other indicators
might well be added for specific studies.
Equally, they are not intended to suggest
essential requirements – it is highly unlikely
that all these indicators will be present,
or even relevant, in any one study.
Many of the quality indicators relate only
to the methods specified in Section II
(i.e. interviews, focus groups, observation
and documentary analysis). For most
appraisal questions, however, quality
indicators that are relevant to other
methods could be added.
The need for professional judgement
The assessment of a qualitative inquiry,
using this framework, will require careful
judgements on the part of the assessor.
These, in turn, will require some knowledge
of qualitative research and some expertise
in using qualitative methods. Judgement
will also be needed in deciding the weight
to attach to particular indicators in order
to assess its ‘fitness for purpose’ – that is,
how well it addresses the objectives for which
it was undertaken. For example, in a study
carried out to evaluate the implementation
of a new scheme, it may well be more
important to have a detailed account of
how practice has affected outcomes, or an
accessibly written report, than to have a
thorough literature review.
Use for assessing proposals
As was noted in Section III, the framework
has been designed to assess outputs from
qualitative inquiry. However, selected
questions and indicators from the framework
could be used to assess proposals for
designing and conducting a qualitative
evaluation (see particularly Qs 2, 5, 6, 7, 9,
10 and 16–18). This, of course, will require
changing from past to present the tense
within which questions are phrased. It is
important to stress, however, that other
questions not included in this framework,
will also be relevant to proposals (for
example, feasibility, timescale, resources,
experience of research team). This framework
is therefore not intended as a comprehensive
aid for proposal assessment.
Qu
al
it
y
in

Qu
al
it
at
iv
e
Ev
al
ua
ti
on
:

A
fr
am
ew
or
k
fo
r
as
se
ss
in
g
re
se
ar
ch

ev
id
en
ce

8
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 8

Literature review (where appropriate)
summarising knowledge to date/key issues
raised by previous research
Aims and design of study set in the
context of existing knowledge/
understanding; identifies new areas
for investigation (for example, in relation
to policy/practice/substantive theory)
Credible/clear discussion of how findings
have contributed to knowledge and
understanding (e.g. of the policy,
programme or theory being reviewed);
might be applied to new policy
developments, practice or theory
Findings presented or conceptualised in a
way that offers new insights/alternative
ways of thinking
Discussion of limitations of evidence and
what remains unknown/unclear or what
further information/research is needed
How has knowledge/
understanding been
extended by
the research?
FRAMEWORK FOR ASSESSING QUALITATIVE EVALUATIONS3
Study being appraised: ...................................................................................
a) Appraisal
questions
b) Quality indicators (possible
features for consideration)
c) Notes on study
being appraised
Findings/conclusions are supported by
data/study evidence (i.e. the reader can
see how the researcher arrived at his/her
conclusions; the ‘building blocks’ of analysis
and interpretation are evident)
Findings/conclusions ‘make sense’/have
a coherent logic
Findings/conclusions are resonant with
other knowledge and experience
(this might include peer or member review)
Use of corroborating evidence to support
or refine findings (i.e. other data sources
have been used to examine phenomena;
other research evidence has been
evaluated: see also Q14)
How credible are the
findings?
FI
N
D
IN
G
S
1
FI
N
D
IN
G
S
2
IV

Content
of
the
framework
9
3 For those wishing to read further about qualitative and evaluative research methods a short list of useful references can be found
at the end.
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 9

Discussion of what can be generalised to
wider population from which sample is
drawn/case selection has been made
Detailed description of the contexts in
which the study was conducted to allow
applicability to other settings/contextual
generalities to be assessed
Discussion of how hypotheses/
propositions/findings may relate to wider
theory; consideration of rival explanations
Evidence supplied to support claims for
wider inference (either from study or from
corroborating sources)
Discussion of limitations on drawing wider
inference (e.g. re-examination of sample
and any missing constituencies: analysis
of restrictions of study settings for
drawing wider inference)
Scope for drawing wider
inference – how well is
this explained?
Discussion of how assessments of
effectiveness/evaluative judgements have
been reached (i.e. whose judgements
are they and on what basis have they
been reached?)
Description of any formalised appraisal
criteria used, when generated and how
and by whom they have been applied
Discussion of the nature and source of any
divergence in evaluative appraisals
Discussion of any unintended
consequences of intervention,
their impact and why they arose
How clear is the basis of
evaluative appraisal?
a) Appraisal
questions
b) Quality indicators (possible
features for consideration)
c) Notes on study
being appraised
Clear statement of study aims and
objectives; reasons for any changes
in objectives
Findings clearly linked to the purposes
of the study – and to the initiative or
policy being studied
Summary or conclusions directed
towards aims of study
Discussion of limitations of study in
meeting aims (e.g. are there limitations
because of restricted access to study
settings or participants, gaps in the sample
coverage, missed or unresolved areas
of questioning; incomplete analysis;
time constraints?)
How well does the
evaluation address its
original aims and purpose?
FI
N
D
IN
G
S
3
FI
N
D
IN
G
S
4
FI
N
D
IN
G
S
5
Qu
al
it
y
in

Qu
al
it
at
iv
e
Ev
al
ua
ti
on
:

A
fr
am
ew
or
k
fo
r
as
se
ss
in
g
re
se
ar
ch

ev
id
en
ce

10
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 10

Description of study locations/areas
and how and why chosen
Description of population of interest
and how sample selection relates to it
(e.g. typical, extreme case, diverse
constituencies etc.)
Rationale for basis of selection of target
sample/settings/documents
(e.g. characteristics/features of target
sample/settings/documents, basis for
inclusions and exclusions, discussion
of sample size/number of cases/setting
selected etc.)
Discussion of how sample/selections
allowed required comparisons to be made
How well defended is
the sample design/
target selection of
cases/documents?
Detailed profile of achieved
sample/case coverage
Maximising inclusion (e.g. language
matching or translation; specialised
recruitment; organised transport
for group attendance)
Discussion of any missing coverage in
achieved samples/cases and implications
for study evidence (e.g. through
comparison of target and achieved
samples, comparison with population etc.)
Documentation of reasons for
non-participation among sample
approached/non-inclusion of selected
cases/documents
Discussion of access and methods of
approach and how these might have
affected participation/coverage
Sample composition/case
inclusion – how well is
the eventual coverage
described?
a) Appraisal
questions
b) Quality indicators (possible
features for consideration)
c) Notes on study
being appraised
Discussion of how overall research
strategy was designed to meet
aims of study
Discussion of rationale for study design
Convincing argument for different
features of research design (e.g. reasons
given for different components or stages
of research; purpose of particular methods
or data sources, multiple methods, time
frames etc.)
Use of different features of design/data
sources evident in findings presented
Discussion of limitations of research
design and their implications for the
study evidence
How defensible is the
research design?
D
ES
IG
N
6
SA
M
P
LE
7
SA
M
P
LE
8
IV

Content
of
the
framework
11
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 11

Description of form of original data
(e.g. use of verbatim transcripts, observation
or interview notes, documents, etc.)
Clear rationale for choice of data
management method/tool/package
Evidence of how descriptive analytic
categories, classes, labels etc. have
been generated and used (i.e. either
through explicit discussion or portrayal
in the commentary)
Discussion, with examples, of how any
constructed analytic concepts/typologies
etc. have been devised and applied
How well has the
approach to, and
formulation of, the
analysis been conveyed?
Description of background or historical
developments and social/organisational
characteristics of study sites or settings
Participants’ perspectives/observations
placed in personal context (e.g. use of
case studies/vignettes/individual profiles,
textual extracts annotated with details
of contributors)
Explanation of origins/history of
written documents
Use of data management methods that
preserve context (i.e. facilitate within case
description and analysis)
Contexts of data sources
– how well are they
retained and portrayed?
a) Appraisal
questions
b) Quality indicators (possible
features for consideration)
c) Notes on study
being appraised
Discussion of:
• who conducted data collection
• procedures/documents used for
collection/recording
• checks on origin/status/authorship
of documents
Audio or video recording of
interviews/discussions/conversations
(if not recorded, were justifiable
reasons given?)
Description of conventions for taking
fieldnotes (e.g. to identify what form of
observations were required/to distinguish
description from researcher
commentary/analysis)
Discussion of how fieldwork methods
or settings may have influenced
data collected
Demonstration, through portrayal and use
of data, that depth, detail and richness
were achieved in collection
How well was the data
collection carried out?
D
A
TA
C
O
LL
EC
T
IO
N
9
A
N
A
LY
SI
S
10
A
N
A
LY
SI
S
11
Qu
al
it
y
in

Qu
al
it
at
iv
e
Ev
al
ua
ti
on
:

A
fr
am
ew
or
k
fo
r
as
se
ss
in
g
re
se
ar
ch

ev
id
en
ce

12
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 12

Use and exploration of contributors’
terms, concepts and meanings
Unpacking and portrayal of
nuance/subtlety/intricacy within data
Discussion of explicit and implicit
explanations
Detection of underlying factors/influences
Identification and discussion of patterns
of association/conceptual linkages
within data
Presentation of illuminating textual
extracts/observations
How well has detail,
depth and complexity
(i.e. richness) of the
data been conveyed?
Clear conceptual links between analytic
commentary and presentations of original
data (i.e. commentary and cited data relate;
there is an analytic context to cited data,
not simply repeated description)
Discussion of how/why particular
interpretation/significance is assigned to
specific aspects of data – with illustrative
extracts of original data
Discussion of how explanations/
theories/conclusions were derived – and
how they relate to interpretations and
content of original data (i.e. how
warranted); whether alternative
explanations explored
Display of negative cases and how they
lie outside main proposition/theory/
hypothesis etc.; or how proposition
etc. revised to include them
How clear are the
links between data,
interpretation and
conclusions – i.e. how
well can the route to
any conclusions be seen?
a) Appraisal
questions
b) Quality indicators (possible
features for consideration)
c) Notes on study
being appraised
Discussion of contribution of sample design/
case selection in generating diversity
Description and illumination of
diversity/multiple perspectives/alternative
positions in the evidence displayed
Evidence of attention to negative cases,
outliers or exceptions
Typologies/models of variation derived
and discussed
Examination of origins/influences
on opposing or differing positions
Identification of patterns of
association/linkages with divergent
positions/groups
How well has diversity of
perspective and content
been explored?
A
N
A
LY
SI
S
12
A
N
A
LY
SI
S
13
R
EP
O
R
T
IN
G
14
IV

Content
of
the
framework
13
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 13

Discussion/evidence of the main
assumptions/hypotheses/theoretical ideas
on which the evaluation was based and
how these affected the form, coverage or
output of the evaluation (the assumption
here is that no research is undertaken
without some underlying assumptions or
theoretical ideas)
Discussion/evidence of the ideological
perspectives/values/philosophies of
research team and their impact on the
methodological or substantive content
of the evaluation (again, may not be
explicitly stated)
Evidence of openness to new/alternative
ways of viewing subject/theories/
assumptions (e.g. discussion of
learning/concepts/ constructions that have
emerged from the data; refinement
restatement of hypotheses/theories in light
of emergent findings; evidence that
alternative claims have been examined)
Discussion of how error or bias may have
arisen in design/data collection/analysis
and how addressed, if at all
Reflections on the impact of the
researcher on the research process
How clear are the
assumptions/theoretical
perspectives/values that
have shaped the form and
output of the evaluation?
a) Appraisal
questions
b) Quality indicators (possible
features for consideration)
c) Notes on study
being appraised
Demonstrates link to aims of
study/research questions
Provides a narrative/story or clearly
constructed thematic account
Has structure and signposting that usefully
guide reader through the commentary
Provides accessible information for
intended target audience(s)
Key messages highlighted or summarised
How clear and coherent
is the reporting?
R
EP
O
R
T
IN
G
15
R
EF
LE
X
IV
IT
Y
&
N
EU
T
R
A
LI
T
Y
16
Qu
al
it
y
in

Qu
al
it
at
iv
e
Ev
al
ua
ti
on
:

A
fr
am
ew
or
k
fo
r
as
se
ss
in
g
re
se
ar
ch

ev
id
en
ce

14
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 14

Discussion of strengths and weaknesses of
data sources and methods
Documentation of changes made to
design and reasons; implications for
study coverage
Documentation and reasons for changes
in sample coverage/data
collection/analytic approach; implications
Reproduction of main study documents
(e.g. letters of approach, topic guides,
observation templates, data management
frameworks etc.)
How adequately has
the research process
been documented?
a) Appraisal
questions
b) Quality indicators (possible
features for consideration)
c) Notes on study
being appraised
Evidence of thoughtfulness/sensitivity
about research contexts and participants
Documentation of how research was
presented in study settings/to participants
(including, where relevant, any possible
consequences of taking part)
Documentation of consent procedures
and information provided to participants
Discussion of confidentiality of data and
procedures for protecting
Discussion of how anonymity of
participants/sources was protected
Discussion of any measures to offer
information/advice/services etc. at end
of study (i.e. where participation exposed
the need for these)
Discussion of potential harm or difficulty
through participation, and how avoided
What evidence is there
of attention to ethical
issues?
ET
H
IC
S
17
A
U
D
IT
A
B
IL
IT
Y
18
IV

Content
of
the
framework
15
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 15

A bibliography of texts addressing quality in
qualitative research is provided in the main
report, but the following are suggested for
those who wish to read further about the
basic principles and nature of qualitative
research and evaluation methods.
Bryman, R. (2001) Social Research Methods,
Oxford: Oxford University Press.
Kvale, S. (1996) InterViews: An Introduction
to Qualitative Research Interviewing,
Thousand Oaks, CA: Sage.
Mason, J. (2002) Qualitative Researching,
London: Sage.
Patton, M.Q. (2002) Qualitative Research and
Evaluation Methods, Thousand Oaks,
CA: Sage Publications Ltd.
Ritchie, J. and Lewis, J. (2003) Qualitative
Research Practice, London: Sage.
Silverman, D. (2000) Doing Qualitative
Research: A Practical Handbook, London: Sage.
V FURTHER READING
Qu
al
it
y
in

Qu
al
it
at
iv
e
Ev
al
ua
ti
on
:

A
fr
am
ew
or
k
fo
r
as
se
ss
in
g
re
se
ar
ch

ev
id
en
ce

16
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page 16

Quality
in
Qualitative
Evaluation
17
The Government Chief Social Researcher’s Office
The Government Chief Social Researcher’s Office (GCSRO) is based in the Prime Minister’s
Strategy Unit and co-ordinates and promotes social research across Government. It
encourages Departments to commission the right research at the right time in order to
promote evidence-based policy making and the effective use of social research. It ensures that
Government research is of the highest quality and uses the most appropriate and up-to-date
methods and techniques. GCSRO helps ensure that the Government social research service has
access to people with the right skills. The office maintains effective links with other professional
groups within Government as well as with the academic community and those engaging in
applied social policy research and evaluation outside Government. Sue Duncan, who is the
Government’s Chief Social Researcher, heads GCSRO. Sue can be contacted by email at
Sue.Duncan@cabinet-office.x.gsi.gov.uk
A web version of the research can be found on Policy Hub (http://www.policyhub.gov.uk).
Policy Hub is a web resource launched in March 2002 that aims to improve the way public
policy is shaped and delivered. It provides many examples of initiatives, projects, tools
and case studies that support better policy making and delivery and provides extensive
guidance on the role of research and evidence in the evaluation of policy.
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page c1

Strategy Unit, Admiralty Arch, The Mall, London SW1A 2WH
Tel: 020 7276 1881
Email: strategy@cabinet-office.x.gsi.gov.uk
Website: www.strategy.gov.uk
The text of this document may be reproduced free of charge in any format or media without requiring specific permission. This is subject
to the material not being used in a derogatory or in a misleading context. The source of this material must be acknowledged as Crown
copyright and the title of the document must be included when being reproduced as part of another publication or service
© Crown copyright 2003
This report is printed on recycled paper produced from at least 75% de-inked post consumer waste, and is totally chlorine free.
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page c2