Automated document-level classification of surveillance and diagnostic colonoscopy for Inflammatory Bowel Disease: An application of natural language processing

Abstract

Background/AIMS: Research utilizing databases is currently limited to administrativelevel data or requires time-intensive chart review. Extensive databases oftext-based medical information have accumulated with the advent of electronicmedical records. However, efficient methods of retrieving this data forclinical or research purposes have not yet been established. Natural LanguageProcessing (NLP) is a process that allows text based searching and documentlevelclassification. NLP based document-level classification may allow patientclassification in domains that are either not available or inaccurate usingstandard diagnosis code classification algorithms. The Automated RetrievalConsole (ARC) is NLP-based software that allows investigators without programmingexperience to design and perform NLP assisted document-levelclassification. ARC works by combining features derived from NLP pipelineswith supervised machine learning. Differentiation of surveillance from diagnosticcolonoscopy is important to identify appropriate colonoscopy practicepatterns for patients with inflammatory bowel disease (IBD) for both researchand quality improvement purposes. Our aim was to test the feasibility andaccuracy of ARC in document- level classification of surveillance and diagnosticcolonoscopy for IBD. Methods: We performed a split validation study of electronic pathologyreports retrieved from patients with IBD from the Michael E. DeBakey VA MedicalCenter in Houston, TX. Patients with IBD were confirmed by chart review.Pathology reports related to colonoscopy performed in these patients wereimported in ARC. The pathology reports were split into two groups: 70% foralgorithm derivation (i.e. 'training') and 30% for validation (i.e. 'testing'). A gastroenterologistreviewed and manually classified pathology reports as eithersurveillance or diagnostic. A model was created within ARC using the trainingcohort and was applied to the testing cohort to classify pathology reports assurveillance or diagnostic. The performance of the model in terms of recalland precision was compared with manual classification of pathology reportsby a gastroenterologist. Results: We screened 382 unique patients with confirmed IBD. A total of 575colonoscopy pathology reports were available on 195 IBD patients, of which400 pathology reports were designated as the training cohort and 175 as thetesting cohort. The best performing classification model of the training cohortin ARC was selected by harmonic mean of recall and precision, or 'F-measure'calculated by cross-validation, with an F-measure of 0.79, recall of 0.80(estimate of sensitivity), and precision of 0.80 (estimate of positive predictivevalue). This model was then applied to the testing cohort to retrievesurveillance colonoscopy reports. Within the testing cohort, a total of 69pathology reports were classified as surveillance by manual review, whereasthe ARC model classified 71 reports as surveillance for a recall of 0.77 andprecision of 0.80. Conclusions: Using ARC we were able to classify surveillance and diagnosticcolonoscopy for IBD without customized software programming. This pilot studydemonstrates that NLP-based document-level classification may be used on largerdatabases to differentiate surveillance from diagnostic colonoscopy in IBD.