Decoding large language models for radiology: strategies for fine-tuning and prompt engineering

Abstract

The advances in large language models (LLMs) have demonstrated sophisticated potential for automating complex tasks within the radiology workflow. From radiology report generation and report summarization to data collection for research trials, these models have proven to be powerful tools. However, optimal implementation of these models requires careful adaptation to the specialized medical domain. In addition, these models tend to generate information that is not truthful or factual, which can adversely affect patient care and clinical decisions. Strategies such as fine-tuning and prompt optimization have been shown to be impactful in eliminating these errors. Although these models undergo rapid updates and improvements, understanding the principles of prompt engineering and fine-tuning provides a foundation for evaluating and maintaining the performance of any LLM deployment. The current article aims to review the recent advancements in radiology using fine-tuning and prompt optimization to leverage LLMs’ capabilities. It delves into various techniques within each strategy, their advantages and limitations, and presents a framework to facilitate the practical integration of LLMs into radiology settings.