The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia

Abstract

Objectives: To investigate the quality of diagnostic and triage advice provided by free website and mobile application symptom checkers (SCs) accessible in Australia. Design: 36 SCs providing medical diagnosis or triage advice were tested with 48 medical condition vignettes (1170 diagnosis vignette tests, 688 triage vignette tests). Main outcome measures: Correct diagnosis advice (provided in first, the top three or top ten diagnosis results); correct triage advice (appropriate triage category recommended). Results: The 27 diagnostic SCs listed the correct diagnosis first in 421 of 1170 SC vignette tests (36%; 95% CI, 31–42%), among the top three results in 606 tests (52%; 95% CI, 47–59%), and among the top ten results in 681 tests (58%; 95% CI, 53–65%). SCs using artificial intelligence algorithms listed the correct diagnosis first in 46% of tests (95% CI, 40–57%), compared with 32% (95% CI, 26–38%) for other SCs. The mean rate of first correct results for individual SCs ranged between 12% and 61%. The 19 triage SCs provided correct advice for 338 of 688 vignette tests (49%; 95% CI, 44–54%). Appropriate triage advice was more frequent for emergency care (63%; 95% CI, 52–71%) and urgent care vignette tests (56%; 95% CI, 52–75%) than for non-urgent care (30%; 95% CI, 11–39%) and self-care tests (40%; 95% CI, 26–49%). Conclusion: The quality of diagnostic advice varied between SCs, and triage advice was generally risk-averse, often recommending more urgent care than appropriate.