CMET-27. NATURAL LANGUAGE PROCESSING FOR THE AUTOMATED QUANTIFICATION OF BRAIN METASTASES IN RADIOLOGY FREE TEXT REPORTS

Abstract

INTRODUCTION: Optimum treatment for patients diagnosed with brain metastases remains a clinical dilemma that depends on the number of metastases present. Whereas a patient with a single metastasis benefits from surgery followed by radiation, a patient with many may not. In this study, we report the use of natural language processing (NLP) to identify the number of metastases present based on free‐text radiology reports in an effort to facilitate high throughput retrospective analysis. METHODS: A total of 2172 radiology reports for patients diagnosed with brain metastases were retrieved. The reports were randomized into six blocks, and each block was manually annotated by two independent reviewers for the number of metastases, using a binary classification (one versus >1 metastasis). The annotated reports were divided into training and test sets. An NLP algorithm using bag‐of‐words technique was developed using the Anaconda distribution (Continuum Analytics) running Python 2.7.1. The classifications of the NLP method were then compared with the manual annotations of our test set, which were based on consensus of human classification. Accu‐racy, precision, recall, and F1 measure were calculated for the NLP method. RESULTS: The manual review was completed in 27 days; mean discordance rate was 36.2%. The overall accuracy of the best logistic regression classifier was 84.6% on the training data and 82.8% on the hold‐out test data. The precision and recall were both 83%, and the F1‐score was 0.83. The algorithm required 28.9 minutes for training, after which it is able to classify the entire dataset in a less than 0.2 second. The model was incorporated into a user‐friendly interface that allows for future report classification. CONCLUSION: NLP is a powerful method for the high‐throughput evaluation of free‐text radiology reports that can have high sensitivity and specificity. Use of NLP can accelerate retrospective clinical research with improved accuracy over manual chart review.