Evaluating the Effectiveness of Large Language Models in Converting Clinical Data to FHIR Format

Abstract

The conversion of unstructured clinical data into structured formats, such as Fast Healthcare Interoperability Resources (FHIR), is a critical challenge in healthcare informatics. This study explores the potential of large language models (LLMs) to automate this conversion process, aiming to enhance data interoperability and improve healthcare outcomes. The effectiveness of various LLMs in converting clinical reports into FHIR bundles was evaluated using different prompting techniques, including iterative correction and example-based prompting. The findings demonstrate the critical role of prompt engineering, with the two-step approach shown to significantly improve accuracy and completeness. While few-shot learning enhanced performance, it also introduced a risk of overreliance on examples. The performance of the LLMs is assessed based on the precision, hallucination rate, and resource mapping accuracy across mammography and dermatological reports from two clinics, providing insights into effective strategies for reliable FHIR data conversion and highlighting the importance of tailored prompting strategies.