promoting access to White Rose research papers

White Rose Research Online


Universities of Leeds, Sheffield and York
http://eprints.whiterose.ac.uk/



This is an author produced version of a paper published in Transforming
Government: People, Process & Policy.


White Rose Research Online URL for this paper:
http://eprints.whiterose.ac.uk/3742/



Published paper

Macintosh, A. and Whyte, A. (2008) Towards an evaluation framework for
eParticipation. Transforming Government: People, Process & Policy, 2 (1).
pp. 16-30.

eprints@whiterose.ac.uk

Towards an Evaluation Framework for eParticipation1

Ann Macintosh* and Angus Whyte+
*The Institute of Communications Studies, The University of Leeds, Leeds, LS2 9JT, UK
+School of Informatics, The University of Edinburgh, Edinburgh, EH8 9LE, UK
Email: A.Macintosh@leeds.ac.uk, awhyte1@inf.ed.ac.uk


Abstract
Purpose
The paper demonstrates the use of a range of perspectives and methods to evaluate
eParticipation initiatives. We argue that there is a need for coherent evaluation frameworks
employing such perspectives and methods, the better to understand current eParticipation
applications and learn from these experiences.
Approach
A case study2 of eParticipation evaluation for four local authority led projects from the ‘top
down’ stream of the UK Local e-Democracy National Project is presented.
Research implications
We argue that further research is needed in two main areas; first, on the applicability of
eParticipation tools to particular contexts, and second, to integrate fieldwork methods to
assess social acceptance of eParticipation and represent the diversity of views obtained from
citizens, community groups and other stakeholders.
Practical implications
The paper describes the application of the framework, demonstrates the importance of a
multi-method approach, and outlines some barriers to using it.
Originality
The described framework is a basis for further development since eParticipation evaluation is
in its infancy despite strong advocacy of evaluation in e-government research and practice
literature.

Classification: case study

Keywords: eDemocracy, digital democracy, eEngagement, e-Government, eParticipation,
Evaluation

1 Introduction

The potential for information and communication technologies (ICTs) to increase political
participation and address the growing democratic deficit across the USA and Europe has long
been the subject of academic debate (e.g. Dutton, 1992). However, it is only in relatively
recent times that there has been sufficient application of ICTs to support democracy that this
‘potential’ could be considered within a real-world context (Weber et al, 2003). The term
‘eDemocracy’ captures both the intent to support democracy and studies of the outcomes and
context. Hacker and van Dijk (2000), using the term ‘digital democracy’ as opposed to
eDemocracy, discuss the emergence of the concept. Previous work (Macintosh 2004a)
considered two components to eDemocracy, one addressing the electoral process including
e-voting, and the other addressing citizen eParticipation in democratic decision-making.

1 Published in: Transforming Government: People, Process & Policy, 2(1), 16-30.
2 This work was undertaken at the International Teledemocracy Centre, Napier University,
Edinburgh, UK

This paper builds on these earlier studies and uses a working definition of eParticipation as
the use of ICTs to support information provision and “top-down” engagement i.e. government-
led initiatives, or “ground-up” efforts to empower citizens, civil society organisations and other
democratically constituted groups to gain the support of their elected representatives.
Effective information provision is often seen as a corollary of effective engagement and
empowerment. Other authors provide similar definitions; see for example Pratchett and
Krimmer (2005).
Nowadays a large range of tools are available to form the basis for eParticipation applications
and it is well accepted that technical, social and political factors need to be considered when
developing eParticipation applications (Mambrey, 2004). However using any type of ICT to
enhance democracy is a challenging task and a number of studies have been conducted to
make the challenges explicit. In 2004 the eGovernment PRISMA project, funded by the
European Commission, considered scenarios on the future use of eParticipation tools
(Kubicek and Westholm, 2005). The resulting paper highlighted a number of issues with
regard to technology design and development. First, the increasing amount of information
available over the internet implies a need for knowledge and information management
systems. Second, the range of stakeholders involved requires personalized communication
integrated with the delivery of relevant information. Third, it highlighted the need for
information systems design to move towards more collaborative working environments to
support government and civil society to work in partnership. The need for a rigorous
evaluation framework was stressed.
A similar future research direction is reported by Macintosh (2006), who arrives at five key
issues for future information systems research to support eParticipation. The first concerns
moving from experimentation and pilots to large-scale usage of eParticipation applications.
The second challenge is concerned with understanding how to design tools to facilitate online
deliberation and support collaborative working environments. The third challenge addresses
the need for reliable representation of the information and analysis of contributions made by
civil society. Fourth, the author stresses the need to embed the technology into political
processes rather than to treat it as an isolated eParticipation exercise. The final challenge,
like Kubicek and Westholm, is concerned with evaluation, so as to understand what has and
has not been achieved.
The aim of this paper is to demonstrate the application of a range of methods and techniques
to evaluate local eParticipation initiatives. The research purpose is to increase understanding
of eParticipation and, consequently, determine how to measure impact and potential
opportunity. Understanding how stakeholders perceive local eDemocracy and how the
technology is used in practice is critical if we are to establish how ICTs may be affecting local
democracy by changing existing practice, and effecting it as they become new instruments for
achieving local democracy.
The approach discussed in this paper was part of a larger study commissioned by Bristol City
Council to evaluate the Local eDemocracy National Project. The national project on local e-
democracy was funded by the then UK Office of the Deputy Prime Minister, as part of a £80m
National Project Program aiming to drive the modernization of local services. The e-
democracy project had the following overarching objectives:
1. To encourage all local authorities to consider the ways in which they can use e-
democracy tools to enhance local democracy and to develop locally appropriate
strategies for implementing such tools where relevant.
2. To ensure that the knowledge and experience of e-democracy that already exists is
systematically exposed and shared across local government for the benefit of all.
3. To develop new tools that support or enhance local democratic practice both within
local government and beyond.
4. To provide a focal point for democratic innovation and the dissemination of best
practice.
5. To begin a sustainable process of electronically enabled participation and
engagement that complements existing democratic structures and processes.

The authors evaluated four of the government-led eDemocracy initiatives. These ‘top-down’
projects were in various states of readiness at the time, and the consequences for the
evaluation are also discussed.

Following this introduction, section 2 overviews the research problem and section 3 explores
methodologies for evaluating eParticipation. Section 4 outlines evaluation of the chosen
eParticipation case studies. Finally, sections 5 and 6 provide an indication for future work and
overall conclusions.

2 Research Problem
The objective of this on-going research is to develop an analytical framework through which
eParticipation initiatives can be evaluated. Although the importance of evaluation of
eParticipation projects is recognised by both government and academia, rigorous evaluations
of eParticipation are hard to find.
In contrast there are a range of methods and the associated studies cited in the literature
concerned with the evaluation of eGovernment (when this term is restricted to the online
delivery of government services). For example, the paper by Sakowicz (2003) provides a
table of examples of government evaluations, presenting the various approaches and giving
the project name, methods used and characteristic features evaluated. One of the most used
tools for the evaluations is the ‘web survey’, which typically consists of a list of questions to
determine the content and the level of online service delivery. Importantly, Sakowicz
concludes that the evaluation analysis for the majority of eGovernment studies is too limited.
The focus tends to be on what is available on the website rather than ‘if and how’ the services
are actually used. The evaluations are concentrated on a particular stakeholder’s perspective,
whether that is the government officials, businesses or citizens. None appear to take a holistic
view and consider all stakeholders. In 2003 a number of researchers from different countries
across Europe undertook an evaluation of the use of technology to support eDemocracy,
focusing on the parliaments and political parties (Trechsel, et al. 2003). However, again, the
evaluation was based on comparative website surveys with some case studies. What the
researchers wished to learn was whether the introduction and diffusion of technology was
having a significant impact upon the practice of democracy. They studied the websites of the
then 15 member states and the 10 candidate states of the European Union, looking at
information provision, user-friendliness, one-to-one interactivity and many-to-many
interactivity. The authors state that this last type of interactivity is the most important since it
has the potential to strengthen the deliberative aspects of citizen participation (p24); however
they go on to point out that having a mechanism available on a website does not mean that it
will be used. This, again, indicates the limitations of current evaluation research and the need
to involve all stakeholders in order to get complete evaluation results.
Recent academic literature on eParticipation has begun to discuss methodological
frameworks for undertaking evaluation. However, although the importance of rigorous
evaluation is recognised, there is little evidence demonstrating the use of such evaluation
approaches in practice. The 2001 study by the OECD considered effective off-line methods
which government could use to engage with citizens and provided a number of ‘guiding
principles’ for off-line citizen engagement. Guiding principle number 9 on ‘evaluation’ stated:
“Governments need the tools, information and capacity to evaluate their performance
in providing information, conducting consultation and engaging citizens, in order to
adapt to new requirements and changing conditions for policy-making.” (OECD 2001,
p.15).
During 2003 the OECD team responsible for citizen engagement commissioned a study into
the potential of ICTs to support citizen engagement in policy-making. The resulting report
made an initial attempt to scope the eParticipation domain from a government perspective,
describe a number of eParticipation case studies in OECD member countries, and highlight
the major issues facing eParticipation. The report highlighted five major issues, including
evaluation. The report stressed that a major challenge was evaluating eParticipation in order
to: make sense of what has, or has not, been achieved; and understanding of how to assess
the benefits and the impacts of applying technology to the democratic process. (Macintosh,
2004b, p89)
However evaluation is novel even in the context of off-line participation. As noted by the
OECD:
“There is a striking imbalance between the amount of time, money and energy that
governments in OECD counties invest in engaging citizens and civil society in public
decision-making and the amount of attention they pay to evaluating the effectiveness
of such efforts.” (OECD 2005, p 10).

This latter report then explains that the ‘evaluation gap’ is due to the relatively recent
realisation by governments of the need to engage with citizens and therefore the evaluation of
public participation is an even newer concept. This despite the OECD report published four
years earlier providing the guiding principle number 9 on ‘evaluation’!

3 Methodological Approaches
As stated earlier, rigorous evaluations of eParticipation applications are hard to find and only
a few embryonic evaluation frameworks have been published (e.g. Whyte and Macintosh,
2003a, Frewer and Rowe, 2005). Forss (2005) suggests three purposes for eParticipation
evaluation: audit; management and learning, and states that in the public sector the
emphasis, typically, has been on audit – where there is a need to monitor the spending of
public money and ascertain whether services are effective and efficient (page 45). However,
given the immaturity of eParticipation work, evaluation for the purpose of learning from current
pilots would seem appropriate. The publication provides a useful source for those wishing to
undertake evaluation, however it does not explore the specifics of eParticipation evaluation
requirements.
In the same OECD publication, Frewer and Rowe (2005) consider some of the practical
issues in evaluating public participation based on their experiences of a number of scientific
and environmental-related citizen engagement exercises. The annex to their paper provides a
list of possible evaluation criteria with suggested questions. This is a useful basis for
constructing an evaluation, but without the necessary detail to evaluate ICT as the media for
the engagement.
The work of Anttirioko (2003) suggests that, in evaluating eDemocracy, the broad capability of
technology to add value should be included. This added value of technology should be
articulated through:
Institutions. To what extent are the ICT-based citizen-centred solutions and
applications integrated in the practices of existing political institutions and how do
they affect actual decision-making processes.
Influence. Are the eDemocracy experiments or practices such that people involved
may truly influence the issues of interest?
Integration. Is the potential of technology used optimally in integrating the elements of
the eDemocratic process, including agenda-setting, planning, preparation, decision-
making, implementation, evaluation and control?
Interaction. Is the potential of technology in disseminating information, facilitating
interaction and conducting political transactions used to increase the transparency,
efficiency, flexibility, cost-effectiveness and inclusiveness of the democratic process?
(p.125).
Gil-Garcia and Pardo (2006) discuss the necessity of a multi-method approach to
eGovernment research, arguing that eGovernment is a complex social phenomenon that can
greatly benefit from the use of multiple disciplines. The case for such a multi-method
approach to eParticipation evaluation is even stronger.
Based on previous work (Whyte and Macintosh, 2003), the authors argue that to evaluate
how effective eParticipation is in engaging a wide audience so as to inform and influence the
policy process, an analytical framework has to be developed that takes into account three
dimensions: the evaluation criteria; the analysis methods available; and the actors involved.
The evaluation criteria, as illustrated in Figure 1, consider three overlapping perspectives:
democratic, project and socio-technical.
• The democratic perspective considers the overarching democratic criteria that the
eParticipation initiative is addressing. Here one of the most difficult aspects is to
understand to what extent the eParticipation affects policy.
• The project perspective looks in detail at the specific aims and objectives of the
eParticipation initiative as set by the project stakeholders.
• The socio-technical perspective considers to what extent the design of the ICTs directly
affects the outcomes. Established frameworks from the software engineering and
information systems fields can be used to assess issues such as usability and
accessibility.

Figure 1 Layered eParticipation evaluation perspectives
Stockholders' aims, methods and
expectations of public engagement
Public take-up,
usefulness and
acceptability
Socio-technical criteria
Project criteria
Democratic criteria
Impact on democracy


A range of data gathering and analysis methods should be used depending, on eParticipation
tools and context of use. These include:
• field observation of relevant actors using the tool in a real-world setting
• interviewing and group discussion with relevant actors
• analysis of online questions and discussion
• analysis of project documentation
• usage statistics from the tools and server logfile analysis
Including methods that analyse interaction ensures that the evaluation provides evidence of
what people actually do with eParticipation tools, as well as what they say they do.
Importantly, using mixed methods allows triangulation of methods and results and, therefore,
helps to maximise their validity.

To understand how stakeholders perceive eParticipation it is also important to involve a range
of actors in the evaluation process. The challenge is to involve each appropriately in the
evaluation.
In government-led eParticipation projects actors might include:
• officials setting up and administrating the eParticipation mechanism
• people who have used the eParticipation mechanism
• others who have not used it
• elected representatives or officials considering the eParticipation results;
• other interested representatives
• project managers and technologists supplying the online tools.
The authors argue that any generalised evaluation framework for eParticipation needs to
state clearly which evaluation criteria are being considered, needs to define the actors
addressed, and needs to ensure that relevant research methods are matched to the
appropriate actor considering the timing, their skills and their willingness to be involved.

4 The Local eDemocracy Evaluation Case Study
This case study is based on work undertaken as part of the UK Local eDemocracy National
Project, specifically “Workstream 4: Public and Stakeholder Opinion and eDemocracy”. The
local eDemocracy projects had originally been classified as top-down projects (which were
government-led and mainly dealt with linking citizens to council processes), ground-up
projects (which are mainly concerned to encourage community networks) and youth-related
eParticipation tools and games. The authors were responsible for the research into ‘top-down’
projects (Whyte, et al, 2005). In all cases the evaluations were constrained by the very limited
time provided for eParticipation to be designed and developed, most of which remained at
formative stages of implementation. The ‘top-down’ projects had to be completed in less than
one year during which the project staff had to be recruited, suppliers contracted, applications
implemented, working practices examined and eParticipation tools publicised and used. The
evaluations in the final months were therefore too short to cover the projects evenly and in
some cases lacked involvement of key actors.
4.1 The Local eDemocracy projects
The four government-led projects discussed in this paper focus on:
1. An ePanel forum for debating city-wide issues;
2. ePetitioning for citizens to lobby their local authority;
3. Personalized survey tool for local authorities to make engagement individually
relevant;
4. Best practice in partnership consultation on crosscutting issues.
Due to space limitations the detailed descriptions of these projects are not presented here but
can be found in Whyte et.al. 2005.
4.1.1 The ePanel
The ePanel project was led by Bristol City Council and had three main aims:
• To demonstrate that the democratic potential of citizens’ panels can be increased using
eDemocracy tools
• To produce detailed guidance for local authorities and case studies demonstrating how
various e-panel approaches can be implemented.
• To strengthen links between authorities engaged in on-line consultation.

The Bristol e-panel website, Askbristol.com integrated various tools such as discussion
forums, surveys and live chats. Registered e-panel members could discuss issues with
experts and with each other before giving their final views in polls or surveys.
4.1.2 ePetitioning
The ePetitioning project was led by The Royal Borough of Kingston upon Thames. e-
Petitioning in the National Project built on the experience of the Scottish Parliament, which
formally launched its e-petitioning system in February 2004 after a 4 year pilot (Adams et al,
2005). The local authority ePetitioner was based directly on this system. Kingston local
authority trialled it in addition to traditional paper petitioning processes. The new service
aimed to increase transparency, accessibility and strengthen the petitioning process through
creating an online service for starting and supporting petitions, and a central location where
petitions could be viewed and their progress tracked.
4.1.3 Personalised survey tool
This project, called Micro Democracy, was led by Swindon Borough Council. It piloted a highly
personalized and localized “micro democracy process” for informing and consulting citizens. It
utilized aspects of customer relationship management and knowledge management. The
stated aims were:
• More efficient and effective consultation in terms of cost per consultation undertaken and
improved response rate;
• Personal engagement about issues that matter to the individual;
• Multi-threaded approach – it was not anticipated that the system would be the sole
method of eParticipation but would be complemented and integrated with other online and
offline approaches.

The developed tool is not a publicly available website but a web-based tool that users within
the local authority may use to generate questionnaires.
4.1.4 Partnership consultation on cross-cutting issues
This project, named “Democractising Cross-Cutting Issues and Partnerships” was led by
Wolverhampton City Council. The aim was not to develop a specific eParticipation tool as
such, but rather to develop best practice guidelines on engaging citizens on issues that cut
across organizational boundaries, and engage them with the work of partnerships of local
public services and civic organisations. To that end a Wolverhampton Partnership website
described the partnership and linked to an ePanel tool for conducting online engagement in a
partnership context.
4.2 Evaluation Design
In order to determine the democratic evaluation criteria, we first needed to understand the
objectives of the Local eDemocracy National Project as a whole. The National Project had
defined overarching ‘democratic criteria’ for what the projects should do to support and
enhance democracy. These are summarized in Table 1 and were used as our democratic
evaluation perspective.

Table 1: Democratic criteria
(Source: Project Initiation Document, National Project on Local e-Democracy v3.0)
Criteria Description
Representation eParticipation should be used to support, complement or enhance
the activities and understanding of representative government, and
should not undermine the value of representative democracy.
Engagement Projects need to support local identity and help individuals
understand, and link into, the wider democratic processes that are
part of their community.
Transparency Projects need to make decision-making processes more
transparent.
Conflict and consensus Projects need to recognise that divergence of opinion may be an
inevitable outcome of enhanced democratic engagement.
Wherever possible, tools should incorporate an expectation of such
divergence and provide opportunities for negotiation, mediation and
consensus building.
Political equality This criterion requires e-democracy to improve the inclusiveness of
policy-making or, at the minimum, not to further disadvantage those
who already are in some way excluded or less powerful in the
political process.
Community control Democracy is about citizens collectively controlling those who take
decisions on their behalf. The tools of e-democracy therefore must
ensure that citizen engagement is closely linked to decision-making
processes and that those who take decisions are responsive to the
communities which they serve.

The project criteria were taken directly from the individual local projects aims and objectives
and involved one or more of the following criteria shown in Table 2.

Table 2: Project criteria
Criteria Description
Engaging with a wider audience Usability criteria such as ‘ease of use’ are important
elements of any evaluation irrespective of there being an
explicit intent to widen participation.. The acceptability
(and hence use) of the technology will be affected by
other non-usability criteria.
Obtaining better informed
opinions
Where the engagement method deliberately provides
respondents with background information in order to elicit
better informed opinions the evaluation should analyse

the use made of this information as an indication of how
relevant it has been.
Enabling more in-depth
consultation
This requires an engagement method that goes further
than simply providing background information by
supporting deliberative debate. The evaluation should
therefore consider analysis of the content and structure of
the discussion to assess the depth achieved.
Cost effective analysis of
contributions
online submission of responses creates opportunities for
more cost effective engagement. Responses made online
save transcription costs, and those to closed questions
can be analysed in real-time.
Providing feedback to citizens where there is an intent to inform participants about the
responses received and their impact on local authority
decisions, the evaluation needs citizens’ assessment of
this feedback.

The socio-technical criteria encompassed aspects of usability, usefulness and acceptability
(Neilson,1993) and aimed for consistency with the Quality Framework for UK Government
Websites (Cabinet Office, 2003). These criteria are described in Table 3.

Table 3: Socio-technical criteria
Criteria Description
Social acceptability
Trust and security Is the information presented accurate, complete and reliable,
and is the information users have provided handled securely?
Relevance and legitimacy Are the intended users satisfied that the tool meets a purpose
relevant to their own and their community’s needs, and are the
content and surrounding processes relevant to that purpose?
Usefulness
Accessibility Is the level of compliance with Web Accessibility Initiative (WAI)
content guidelines sufficient to meet the needs of users with
disabilities?
Appeal Is the take-up in line with expectations, and do the intended
users like it enough to want to use it?
Content clarity Can users understand what the content means in relation to
their task or situation?
Responsiveness Does the tool and/or process answer the user’s questions
quickly and effectively?
Usability
Navigation and
organisation
Do the intended users have sufficient and consistent
information about their current position within the site
organisation, the path taken, and options available?
Efficiency and flexibility Can the intended users perform tasks in an acceptable time,
and are there appropriate short-cuts for doing repetitive or
familiar tasks?
Error recovery Can the intended users ‘undo’ their previous action, and are
they guided effectively on the correct procedure so they can
continue without distraction or hesitation?

During the evaluation we targeted these actors:
(a) Citizens who have used the e-democracy tools deployed (or agreed to take part in a
pilot).
(b) Citizens who have not used the tools.
(c) Councillors involved in the engagement process.
(d) Engagement ‘owners’: managers responsible for aspects of the engagement process.
(e) Project managers/ technologists, whether employed by the Council or by suppliers.

(f) ‘Internal’ users: moderators or administrators.

Also, we used a range of methods:
1. Semi-structured interviews were carried out face-to face and supplemented with telephone
interviews. These were audio-recorded and notes taken to enable quicker analysis.
2. Field tests of the tools involved observing how members of the public interacted with the
tools, to better understand their acceptability and usability in a real-world setting (e.g.
public libraries).
3. Online questionnaires to gather perceptions of the tools acceptability
4. Project documentation was inspected to understand each project’s rationale and
expectations.
5. Results of online discussions were analysed.
6. Web server log analysis provided information on visits and page requests.

To appreciate how each of the actors should be involved and plan appropriate questions, the
‘key dimensions’ of eParticipation adapted from previous work to help characterize
eParticipation initiatives (Macintosh, 2004b) was used. Table 4 shows these key dimensions
(in bold) along with the evaluation questions, the actors targeted and the methods used.

Table 4: Detailed evaluation questions and how they were addressed
Dimension/ Questions Methods Actors
Type of engagement
How does the project relate to the Council’s e-Democracy and
participation agenda, and how should it benefit relations with the public,
elected representatives (executive and opposition), partners and
national government? Have views changed in light of the outcomes
apparent so far? If so, how?
1 all
What are actors’ views of the weight given to engagement results in the
Council’s decision-making, and how do their perceptions differ from
those of officials & representatives who ‘own’ the engagement process?
1 all
Stage in decision-making
Where do actors think the engagement process fits into service delivery
and/or more general policy making (as ‘monitoring’, ‘agenda setting’
etc.)? How does this vary according to the issue or topic?
1 all
What important differences, if any, are there between the issues raised
in online engagement and those using other methods?
1, 2, 3, 5 all
Actors
Who are the relevant actors (politicians, officials, technologists, citizens
as individuals and groups), and why are they involved? i.e. who does
the work, who controls it, who are the actual or potential beneficiaries,
who may be adversely affected, who is otherwise involved?
1, 3, 5 all
What important differences, if any, are there between the roles of the
moderator (or facilitator) in online engagement compared with other
approaches used?
2 (d), (e),
(f)
Technologies used
How are the tools provided used? 1, 2, 3, 5,
6
all
How useful are the tools to their intended users. all (a), (b),
(f)
Rules of engagement
What is the impact on public engagement of the project and what
implications does that have for the engagement process?
all all
How do actors view the privacy implications of engagement and what
measures have/should be taken to address these (considering DP &
FOI)?
1, 2, 3 all
Duration & sustainability

How does the project impact on other public engagement activities,
especially when they also have an online element?
1, 2, 4, 5 all
4.3 Evaluation Results
There is scope here to outline what we concluded from applying the evaluation framework,
the results of which were fed back to the local authorities and technology providers involved
(Whyte et al 2005).
Representation
The major strength of Bristol City Council’s e-panel was its leverage of existing liaison
between the consultation team and service departments, enlisting their direct participation in
online discussion. The Royal Borough of Kingston’s e-petitioner project had strong support
from councillors, although the outcomes of e-petitions remained uncertain and there were
weaknesses in the integration with other engagement processes. The Personalised Survey
project in Swindon generated keen interest from councillors. The Wolverhampton Partnership
rested on a strong consultation infrastructure, although support from partners and councillors
appeared limited.
Engagement
In all cases strong efforts were made to encourage public response on issues of local
relevance. The early evidence was that this was forthcoming, but only when the issues were
general enough to affect a broad cross-section of citizens.
Transparency
The ePetitioner project was strongest on this point since it established a process for
publishing decision outcomes. There was potential in each project to enhance transparency,
but it would have been preferable for them to first establish what citizens would need or
expect in terms of enhanced transparency. This is necessary because “enhancing
transparency” may be taken to mean either providing detailed information or hiding it in the
name of simplicity. The projects each had published policies on privacy and acceptable use,
with the exception of the Micro Democracy personalised survey project where these were
strengthened on our recommendation.
Conflict and consensus
Each of the projects provided an online forum and opportunities for divergence of opinion on
the issues raised and the method for raising them, with the exception of the Personalised
Survey project. The preparation for effective moderation of such discussions was a strong
feature of the e-panel project and the Wolverhampton Partnership project, although
Kingston’s preparations for moderating any controversial e-petitions were not fully developed
at the time.
Political equality
The projects each showed strong potential for greater inclusiveness. There was evidence that
the tools were already being actively used by hundreds of citizens in each of the local
authority areas. There were also signs that these were mostly not previously “engaged” in
local authority decision making. Demographically there were disabled and minority ethnic
users almost in proportion to local populations, but male and middle-aged users
predominated. The Personalised Survey project had strong potential given its integration of
online and offline channels, although its take-up could not be assessed during the evaluation
period.
Community control
Citizens had modest expectations that their views would have some impact on decision-
making and strong expectations that the councils should in any case publish a response to
their input. The evidence that citizens were satisfied with the arrangements was limited but
mostly positive for the ePetitioner and e-panel projects. The Wolverhampton Partnership
showed strong potential to liaise effectively with existing community groups, although it was
not possible to directly assess the latter’s involvement in the time available. The personalised
Survey project placed much emphasis on responsiveness, though again citizens’ views on
that were unavailable.

5 Future work
This paper has demonstrated the use of a range of perspectives and methods to evaluate
eParticipation initiatives and provide a coherent evaluation framework for their integration and
use. However, at the same time, the authors recognise that there is a need to develop further

the framework which recognises the complexity of the domain and the need to use methods
relevant to all those with an interest in the outcome.
The evaluation framework presented here seeks to integrate different perspectives on
evaluation criteria, a range of stakeholders and methods for assessing eParticipation projects.
The aim is to evaluate eParticipation applications in terms of their impact on local democracy,
their planning of public engagement and quality of the tools provided for public engagement.
The described framework is a basis for further development since eParticipation evaluation is
in its infancy despite strong advocacy of evaluation in e-government research and practice
literature.
This framework, has been the subject of academic debate within the Demo-Net project which
is a Network of Excellence on eParticipation funded by the European Commission. The
framework is viewed by the members of the network as a ‘reasonable’ starting point on which
to build a fully multi-method approach to eParticipation evaluation.
The evaluation highlighted measures taken to ensure that ease-of-use and accessibility were
not barriers to take-up. However, the main barriers to citizens making effective use of the
tools are much more likely to be the organizational ones of coordinating the provision of the
tools on the one hand, and, on the other hand, societal ones of citizens and civic groups
accepting and using them. Evaluation research in the Information Systems field (see e.g.
Hirschheim, 2007) provides a variety of relevant theoretical perspectives that may usefully be
applied to eParticipation to build on the few existing studies (e.g. Whyte and Macintosh,
2003b).

6 Conclusions
Applying the eParticipation framework resulted in an assessment that was considered fair by
participants. However the framework needs further development. More consideration needs
to be placed on how and when to use tools in which contexts, also, on how to combine tools
to enable inclusive engagement. The evaluation timescale meant important aspects of the
framework were downplayed.
In terms of practical application, local authorities are still relying on questionnaires that deal
with user satisfaction levels to assess whether local democracy is being enhanced by
eParticipation. There is a strong case for using field study methods to observe and analyse
eParticipation tools being used in community group settings and public places. This would
complement views expressed in individual discussions and group workshops, to assess social
acceptability in a better way.
Evaluation is vital to providing a well-rounded and holistic view of any eParticipation initiative
or project. It needs to be independent of the project, yet planned at the beginning as part of
the project. It should be critically honest about what things have gone well, as well as about
its difficulties and shortcomings. To achieve this we have argued that a range of qualitative
methods are needed that do not simply relate to user satisfaction but rather establish how the
eParticipation project is affecting local democracy by changing existing practice, and effecting
it as they become new instruments for achieving local democracy. We should note however
that there are no standard definitions of effectiveness in eParticipation, nor should we expect
any to emerge. As we have stated the evaluation must accommodate a variety of subjective
assessments of the many actors involved, and of the many constituent elements of the
eParticipation application. eParticipation is a hybrid of various technologies, social and
political measures and there is a need to improve understanding of the relationships between
these components and how their respective evaluation practices can be applied to
eParticipation as a whole.

7 References
Adams, N. J., Macintosh, A., and Johnston, J. (2005); 'e-Petitioning: Enabling Ground-up
Participation'; Challenges of Expanding Internet: E-Commerce, E-Business and E-
Government; Matohisa Funabashi and Adam Grzech (eds); 5th IFIP Conference on e-
Commerce, E-Business and E-Government (I3E'2005); October 26-28 2005, Poznan,
Poland, pp265-279.
Anttirioko, Ari-Veikko (2003). ‘Building Strong e-Democracy – The Role of Technology in
Developing Democracy for the Information Age’, Communications of the ACM, Vol. 46 No.
9.
Cabinet Office (2003) Quality Framework for UK government website design available at:

http://archive.cabinetoffice.gov.uk/e-government/resources/quality-framework.asp
Dutton, W. H., (1992) “Political Science Research on Teledemocracy” Social Science
Computer Review, Vol. 10 No. 4, pp.505-22.
Forss (2005) In Evaluating Public Participation in Policy Making. OECD, Paris.
Frewer L.J. and Rowe R. (2005) ‘Evaluating Participation Exercises: Strategic and Practical
Issues’. In Evaluating Public Participation in Policy Making. OECD, Paris.
Gil-Garcia and Pardo (2006) “Multi-Method Approaches to Digital Government Research:
Value Lessons and Implementation Challenges”. Proceedings of the 39th Annual Hawaii
International Conference on System Sciences (HICSS-39), 2006.
Hacker and van Dijk (2000) ‘What is Digital Democracy?’. In Hacker, K.L and Jan van Dijk.,
(eds) Digital Democracy issues of theory and practice. Sage Publications.
Hirschheim, R. (2007) “Introduction to the Special Issue on ‘Quo Vadis TAM – Issues and
Reflections on Technology Acceptance Research’” Journal of the AIS, Volume 8 Issue 4
Article 18 April, 2007.
Hoff, J. (1999) “Towards a Theory of Democracy for the Information Age” paper presented at
Virtual Society? Democracy Platform UK-Nordic Meeting Copenhagen, 16-17 September
1999.
Kubicek, H., and Westholm, H. (2005) Scenarios for Future Use of eDemocracy Tools in
Europe, International Journal of Electronic Government Research, 1(3), p33-50, July-
September 2005.
Macintosh, A. (2004a) “Characterizing E-Participation in Policy-Making”. In Proceedings of the
37th Annual Hawaii International Conference on System Sciences (HICSS-37), January 5
– 8, 2004.
Macintosh, A. (2004b) “Using information and communication technologies to enhance citizen
engagement in the policy process” in Promises and Problems of E-Democracy:
Challenges of online citizen engagement. Paris: OECD. ISBN 92-64-01948-0.
Nielsen, J. (1993) Usability Engineering. Academic Press, Boston, USA.
OECD (2001) Citizens as Partners: Information, consultation and public participation in policy-
making: OECD, Paris.
OECD (2005). Evaluating Public Participation in Policy Making. OECD, Paris.
Pratchett, L. and Krimmer, R. (2005) “The Coming of e-Democracy”, International Journal of
Electronic Government Research Vol. 1 No. 3 (Special issue on electronic democracy)
Sakowicz, M. (2002). How to Evaluate e-Government? Different Methodologies and Methods.
Available at
unpan1.un.org/intradoc/groups/public/documents/NISPAcee/UNPAN009486.pdf.
Trechsel, A. H., Kies. R., Mendez F. et al (2003) Evaluation of the use of new technologies in
order to facilitate democracy in Europe. University of Geneva, Available at
c2d.unige.ch/int/OverviewInstits/Main_Report_final%201.pdf.
Weber, L., Loumakis, A., and Bergman, J. (2003) “Who Participates and Why? An Analysis of
Citizens on the Internet and the Mass Public” Social Science Computer Review; Volume
21, (1), pp.25-32.
Whyte, A. and Macintosh, A. (2003a) Analysis and Evaluation of e-consultations; e-Service
Journal Volume 2, No 1, 2003.
Whyte, A. and Macintosh, A. (2003b) ‘Representational politics in virtual urban places’
Environment and Planning A, Vol.35, pp. 1607-1627.
Whyte, A., Renton, A. and Macintosh, A. (2005) eDemocracy from the Top Down: An
Evaluation of eDemocracy Activities Initiated by Councils and Government Published by
Bristol City Council for The Local eDemocracy National Project. Available at:
http://itc.napier.ac.uk/ITC/Documents/eDemocracy_from_the_Top_Down_ODPM_2005.pd
f.