A prospective pragmatic evaluation of automatic trial matching tools in a molecular tumor board

Abstract

Publicly available trial matching tools can improve the access to therapeutic innovations, but errors may expose to over-solicitation and disappointment. We performed a pragmatic non-interventional prospective evaluation on sequential patients at the Molecular Tumor Board of Centre Leon Berard. During 10 weeks in 2024, we analysed 157 patients with four clinical trial matching tools from the 19 screened: Klineo, ScreenAct, Trialing and DigitalECMT. Each patient had 2.19 trials proposed on average, and 38% had no trials suggested. The mean performances were precision = 0.33, recall = 0.32, AP@3 = 0.45, and NDCG@3 = 0.34. Using all the tools can increase to 26% the clinical trial options. The most frequent error concerned the type of gene variants required by the selection criteria. We showed that using a Large Language Model on the patients’ molecular reports could improve the performance by up to 5%. We recommend that experts supervise the results and we advocate for improved technologies.