Ushahidi Haiti Project Evaluation 1 Independent Evaluation of the Ushahidi Haiti Project Nathan Morrow Nancy Mock Adam Papendieck Nicholas Kocmich 8 April, 2011 DISI ��� Development Information Systems International Ushahidi Haiti Project

Ushahidi Haiti Project Evaluation 2 Table of Contents TABLE OF CONTENTS .......................................................................................................................................2 ACRONYMS .............................................................................................................................................................3 EXECUTIVE SUMMARY .....................................................................................................................................4 1 INTRODUCTION AND METHODS ...........................................................................................................8 2 RELEVANCE ................................................................................................................................................12 3 EFFECTIVENESS ........................................................................................................................................13 4 EFFICIENCY ................................................................................................................................................18 5 IMPACT .........................................................................................................................................................25 6 SUSTAINABILITY ......................................................................................................................................27 7 RECOMMENDATIONS ..............................................................................................................................29 8 APPENDICES ...............................................................................................................................................32 APPENDIX 1: INTERVIEWS ....................................................................................................................................32 APPENDIX 2: DRAFT OF TIMELINE AND EVENTS ................................................................................................. 33 APPENDIX 3: POTENTIAL EXAMPLES OF UHP IMPACT ........................................................................................ 34 This report was produced by a team of independent consultants for the Ushahidi Haiti Project. These opinions have not been approved by the Ushahidi Haiti Project or Tufts University and should not be taken to represent their official position. The Ushahidi Haiti Project does not guarantee the accuracy of the data included in this study, nor does it accept responsibility for any use made thereof.

Ushahidi Haiti Project Evaluation 3 Acronyms AEA DISI American Evaluation Society Development Information Services International EIS Emergency Information System GPS Global Positioning System ICRC International Committee of the Red Cross IFRC International Federation of Red Cross and Red Crescent Societies NGO Non-governmental Organization OCHA United Nations Office for the Coordination of Humanitarian Affairs PaP SMS UHP Port au Prince Short Message Service Ushahidi Haiti Project UN United Nations USAID USCG United States Agency for International Development United States Coast Guard USMC WFP United States Marine Corps World Food Programme

Ushahidi Haiti Project Evaluation 4 Executive Summary The Ushahidi Haiti Project (UHP) was a volunteer-driven effort to produce a crisis map after the January 12, 2010 earthquake in Haiti. The project represents an impressive proof of concept for the application of crisis mapping and crowdsourcing to large scale catastrophes and a novel approach to the rapidly evolving field of crisis informatics. This evaluation was commissioned by the student group at the Tufts University Fletcher School who instrumental in the UHP deployment and is intended to be a learning evaluation as opposed to an accountability evaluation. The evaluation���s purpose is to serve the needs of UHP users and was structured around the Organization of Economic Development���s (OECD) criteria: relevance, effectiveness, efficiency, impact and sustainability. The evaluation team utilized a mixed methods approach combining surveys of key user and volunteer groups, 30 interviews with core UHP volunteer staff and stakeholders, document review/data analysis primarily focused on UHP message data, volunteer chat forums, coordination chat history and key news sources/blogs. Relevance To what extent does UHP address unmet needs of beneficiaries, humanitarians and the donor system which supports it? Looking at the relevance of a project gets at the heart of whether or not it was a good idea from the start. The enormous appeal of the UHP in the media and among stakeholders can be largely attributed to its profound relevance in early response to emergencies. The UHP addressed key information gaps (1) in the very early period of the response during the first days and weeks post-quake before UN and other large organizations were operational, (2) by providing situational awareness and critical early information with a relatively high degree of geographic precision, (3) by providing situational information for smaller NGOs that did not have a field presence in Haiti, (4) by helping smaller, privately funded responses to more appropriately target needs and, (5) by facilitating private citizen actors. The UHP also was relevant in the sense that it directly engaged affected Haitians and the Haitian Diaspora in the articulation of need and the organization of local capacity for response. Vigilant attention to broad concerns related to the protection of participants and vulnerable individuals is key to maintaining and improving relevance to the willing participants in open, social crisis mapping systems such as UHP. Effectiveness To what extent did responders actually make decisions based upon UHP and the information it provided? The question of information use is central to the evaluation of information interventions. However, the linkage between information and response in general is typically tenuous at best, and the evaluators were not surprised that findings regarding such linkages were mixed. Perhaps the most common use of information aggregated by UHP was for situational awareness. The Department of State analysts for the USG interagency task force used Ushahidi in at least one case to help triangulate conclusions about the situation on the ground, and US military organizations used Ushahidi data feeds along with other sources in a similar manner to inform their early situational assessments. There is also some evidence of the information being used for specific operational and tactical actions targeting specific communities (and to a much lesser extent, individuals). US marines used the information to identify ���centers of gravity��� for deployment of field teams to areas of need, for example. Likewise, small privately-funded nonprofits without prior field presence in Haiti used the information to identify institutions such as orphanages or hospitals as possible partners. For example, the organization NYC Medics were able to identify the Albert Schweizer Hosptial as an institution with capacity to use the doctors and supplies that the organization was able to mobilize. There is also evidence that the volunteer geo-location services offered by the UHP core team were useful for SAR efforts, for example through the resourceful geo-coding efforts of Anna Schultz at Tufts, among others. This team

Ushahidi Haiti Project Evaluation 5 and its volunteer leadership, like Patrick Meier, were also effective in recognizing and catalyzing linkages and collaborations with other key systems and networks such as Mission 4636, Crowdflower, and OpenStreetMap which collectively added tremendous value to the overall stream of crisis information. There is less evidence in the data sources reviewed by evaluators that the UHP web application itself was used extensively for soliciting additional information and feedback on individual reports, or status tracking and the monitoring of individual incidents over time, though this was indeed happening through volunteer efforts to some extent. Why was the information used? The UHP information was used primarily because it was the only map aggregator of information coming from the affected area during the early days after the quake. Again, this is a testament to the high degree of relevance of the UHP project. The credibility of the project and project team was often cited as a reason for the continued use of the information, and high levels of trust built through common graduate academic programs and pre-existing professional networks such as the International Network of Crisis Mappers cannot be underestimated. Why was the information NOT used? Barriers to use of the UHP were often significant, if also largely unsurprising. Primary among these barriers was a general inconsistency of the dynamic ���event data��� aggregated and syndicated by UHP with the specific and often relatively rigid information requirements of traditional responding organizations which typically require certain types of information at certain times and organized around certain response sectors and geographies. The UHP team indeed made efforts to adapt to these requirements but it is still cited as a significant obstacle to use throughout the early response. Information overload remains an issue in general for these responders. Use was also limited due to apparent low awareness of the project within the humanitarian community in Haiti, along with low knowledge of and capacity to use the crowdsourced information. While a clear strength of the UHP was its healthy cooperative relationship with other crisis informatics initiatives like Mission 4636, it is likely that this may have also obscured a distinct UHP ���corporate identity,��� and thereby negatively impacted awareness among groups of potential users initially less familiar with Ushahidi. Interviews also revealed some general ���suspicion of the crowd��� and related questions about the representativeness and quality of the data. Finally, there were several technological limitations to information use. USG staff cited outdated computers, browsers as well as internet communication security policy as significant obstacles to accessing the UHP website and data streams. Limited bandwidth was cited by organizations on the ground in Haiti. Efficiency How efficiently did UHP add value through the processing and mapping of reports? Estimates of 40,000 to 60,000 reports were processed through UHP/Mission 4636, and 3,584 events have been mapped in Haiti. Of these, 80% were mapped in the first month and 72% of all points were mapped in Greater PaP. UHP leveraged some tremendously efficient crowdsourcing strategies to map a translated and geo-coded stream of data, namely the crowdsourced mapping of Haiti using the OpenStreetMap and the crowdsourced translation of Mission 4636 text messages that was eventually connected to the volunteers of UHP with the support of CrowdFlower. Open Street Map and Mission 4636 created improved open source maps and translated messages respectively, and UHP relied heavily on these innovations as primary sources of information behind the UHP site���s dynamic map. The translation and geo-coding of messages in preparation for reporting in and of itself was fast, though there is evidence that there were occasional delays between steps in the system and the ultimate mapping of reports. Often the message detail was not sufficient to correspond to specific relief planning needs related to the number of people in need and their location. Duplicate messages indicated some technical or systemic problems that were not corrected by quality assurance efforts. At certain phases, uneven capacity of volunteers and insufficient efforts to build consistent capacity or implement more rigorous quality assurance also negatively impacted the value of classification and in some cases the accuracy of locations. Although there were some