A systematic review of methods used for COVID-19 telehealth systems evaluation: lessons from past experiences for future use

Abstract

Background: During the COVID-19 emergency, telehealth systems were rapidly designed and integrated into healthcare delivery frameworks. The main question is whether these systems have been adequately evaluated and whether they are worthy of entering the health service cycle. Thus, this study’s objective was to critically analyze the literature on remote health systems developed in response to COVID-19, focusing specifically on the evaluation approaches used. Methods: The present investigation was executed in 2024, focusing on literature published over five years following the onset of the COVID-19 pandemic, extending until January 2024. Comprehensive searches were conducted across PubMed, Web of Science, and Scopus databases. A team of four researchers systematically evaluated and critically appraised relevant articles. Subsequently, the extracted data underwent rigorous analysis, comparison, and reporting to distill key findings. Results: Overall, 26 articles met the inclusion criteria. The results indicated that the predominant objective of evaluations was to examine system performance (n = 8, 30.77%). Also, the methodology used in most evaluations was Qualitative and quantitative (non-integrated) (n = 18, 69.23%). In terms of evaluation period, most evaluations were summative (n = 12,46.15%), and the evaluator population in most systems was the healthcare team (n = 14, 53.85%). Conclusions: Given the COVID-19 emergency conditions, evaluations of telehealth systems have not been conducted systematically and with adequate methods, tools, and populations. Therefore, appropriate systems should be designed and adequately evaluated for future epidemics. Ultimately, the proactive and comprehensive design and evaluation of telehealth systems will create infrastructures that can respond effectively and sustainably to future outbreaks.