PART I: CONCEPTUAL FRAMEWORK

Abstract

Preface Why this Handbook? e overall aim of this handbook is to provide a practical guide on the evaluation of eHealth systems. Over the years, we have seen a steady growth in the number and type of eHealth systems being adopted in different healthcare settings. Proponents of these systems claim eHealth can improve the quality of care provided, leading to better provider performance and health outcomes. Yet the evidence for such claims is mixed thus far, with some studies demonstrating benefits, others showing little to no impact, and some settings being even worse off than before. Understandably, there are now increasing pressures on government agencies and health organizations to demonstrate tangible return on value for the significant eHealth investments made. Despite the growing importance and need to evaluate eHealth systems, there are relatively few formal courses available from post-secondary educational institutions on how to plan, conduct, report and appraise eHealth evaluation studies. Most educational institutions that offer degree programs related to health research, administration and services would typically include eHealth evaluation as part of their health research methods or program evaluation courses. Of those that offer health informatics degree programs, only some have eHealth evaluation as a full self-contained course. For institutions that offer eHealth evaluation as either a course or a topic within a course, the choice of textbooks and reference materials can vary greatly depending on what is available and the preference of the instructors. To date, there have been just a few books published on eHealth evaluation. Notable examples are the reference texts edited by van Gennip and Talmon (1994), Anderson and Aydin (2005), and Friedman and Wyatt (2006), as well as the handbook written by Brender (2006). Aside from these, we are not aware of other major reference texts published in the last 10 years focused solely on this topic. Yet during this period we have witnessed an exponential growth in the number of published journal articles and government reports on eHealth evaluation. ese publications often contain descriptions of different evaluation approaches and/or field studies on the design, implementation, use and effects of particular eHealth systems in specific settings. Overall, what seems lacking is a reference text that brings together these diverse approaches, studies and lessons as a coherent body of literature on the current state of knowledge in eHealth evaluation in a manner that is both rigorous and practical. With the increasing use of eHealth systems and the growing demand to demonstrate their value, there is a strong case to be made to incorporate eHealth evaluation as part of the adoption process in order to generate the empirical evidence needed. Given the lack of current reference texts on eHealth HANDBOOK OF vi EHEALTH EVALUATION evaluation, we believe it is both necessary and timely to publish an up-to-date resource that can help those involved with eHealth evaluation in healthcare settings. Rather than publishing an academic textbook in the traditional manner, we have opted for a handbook in the form of a freely available electronic book or e-book. Compared to a conventional text, we believe such a freely available e-book can better serve as a more flexible, updatable and practical guide for those who need to plan, conduct, report and appraise eHealth evaluation in the field setting. Who is it for? is handbook is intended as a primary resource or a supplementary resource to textbooks on eHealth for students enrolled in courses related to eHealth evaluation. is handbook is also intended for individuals who are involved with the planning, design, implementation, use, support and assessment of eHealth systems in different healthcare settings. ese individuals may be managers, analysts, developers, providers and trainees who are involved with some aspects of eHealth systems as part of their day-to-day work. In large organizations some of these individuals may have dedicated roles in eHealth evaluation. But often we expect them to be responsible for aspects of eHealth planning, design, implementation and support, with evaluation assigned as an afterthought or an adjunct role on the side. e varied audience identified above suggests that this e-book is written for individuals who are not experts in eHealth evaluation but are expected to engage in such assessment activities in their own workplaces. In fact, much of the content in this handbook can be considered introductory in nature. is is to ensure those who are relatively new to the subject can gain a basic understanding of the current state of eHealth evaluation approaches, studies and findings, and can see how this knowledge could be applied and interpreted within their own settings. At the same time, this handbook can also be a useful resource for individuals who are already familiar with eHealth evaluation. In particular, the handbook provides a systematic overview of the different evaluation approaches with case examples that have been applied and reported for a wide range of eHealth systems across different healthcare settings. As such, the handbook can serve as a reference text on details regarding particular evaluation approaches and the current state of knowledge in selected eHealth domains covered as case examples. Francis Lau and Craig Kuziemsky Editors Contents List of Tables and Figures ............................................................................................ iii Preface ..............................................................................................................................v Acknowledgements .................................................................................................... viii Introduction What is eHealth? ............................................................................................................1 Francis Lau, Craig Kuziemsky What is eHealth Evaluation? .......................................................................................3 Francis Lau, Craig Kuziemsky What is in this Handbook? ..........................................................................................5 Francis Lau, Craig Kuziemsky Part I: Conceptual Foundations 1. Need for Evidence, Frameworks and Guidance .............................................11 Francis Lau 2. Benefits Evaluation Framework ........................................................................23 Francis Lau, Simon Hagens, Jennifer Zelmer 3. Clinical Adoption Framework .......................................................................... 55 Francis Lau, Morgan Price 4. Clinical Adoption Meta-Model ........................................................................77 Morgan Price 5. eHealth Economic Evaluation Framework .................................................... 93 Francis Lau 6. Pragmatic Health Information Technology Evaluation Framework ...... 109 Jim Warren, Yulong Gu 7. Holistic eHealth Value Framework ................................................................123 Francis Lau, Morgan Price Part II: Methodological Details 8. Methodological Landscape for eHealth Evaluation .................................. 145 Craig Kuziemsky, Francis Lau 9. Methods for Literature Reviews ..................................................................... 157 Guy Paré, Spyros Kitsiou 10. Methods for Comparative Studies ..................................................................181 Francis Lau, Anne Holbrook 11. Methods for Descriptive Studies ....................................................................199 Yulong Gu, Jim Warren HANDBOOK OF EHEALTH EVALUATION 12. Methods for Correlational Studies .................................................................213 Francis Lau 13. Methods for Survey Studies ............................................................................227 Francis Lau 14. Methods for eHealth Economic Evaluation Studies .................................. 243 Francis Lau 15. Methods for Modelling and Simulation Studies .........................................261 James G. Anderson, Rong Fu 16. Methods for Data Quality Studies .................................................................277 Francis Lau 17. Engaging in eHealth Evaluation Studies .......................................................291 Craig Kuziemsky, Francis Lau Part III: Selected eHealth Evaluation Studies 18. Value for Money in eHealth: Meta-synthesis of Current Evidence ........ 307 Francis Lau 19. Evaluation of eHealth System Usability and Safety .................................... 337 Morgan Price, Jens Weber, Paule Bellwood, Simon Diemert, Ryan Habibi 20. Evaluation of eHealth Adoption in Healthcare Organizations .................351 Jim Warren, Yulong Gu 21. Evaluation of Picture Archiving and Communications Systems .............365 Don MacDonald, Reza Alaghehbandan, Doreen Neville 22. Evaluation of Provincial Pharmacy Network ...............................................383 Don MacDonald, Khokan C. Sikdar, Jeffrey Dowden, Reza Alaghehbandan, Pizhong Peter Wang, Veeresh Gadag 23. Evaluation of Electronic Medical Records in Primary Care: A Case Study of Improving Primary Care through Information Technology ................397 Lynne S. Nemeth, Francis Lau 24. Evaluation of Personal Health Services and Personal Health Records ... 411 Morgan Price, Paule Bellwood, Ryan Habibi, Simon Diemert, Jens Weber 25. Evaluating Telehealth Interventions ............................................................. 429 Anthony J. Maeder, Laurence S. Wilson Part IV: Future Directions 26. Building Capacity in eHealth Evaluation: e Pathway Ahead ...............447 Simon Hagens, Jennifer Zelmer, Francis Lau 27. Future of eHealth Evaluation: A Strategic View ..........................................461 Francis Lau Glossary...................................