Evaluation of automatic speech recognition prototype for estonian language in radiology domain: A pilot study

Abstract

The aim of this study was to determine the dictation error rates in finalized radiology reports generated with a new automatic speech recognition (ASR) technology prototype for the Estonian language. For training a language model, 177 659 real radiology reports from different imaging modalities were used. Manually normalized versions of 1299 randomly selected reports were created to standardize the report corpus. The ASR prototype, incorporating the trained language and acoustic models, was tested in Radiology Department, North Estonia Medical Centre, Tallinn, Estonia, by 17 radiologists (11 female and 6 male). In total, 424 reports were dictated, including 77 067 x-ray, 30 929 ultrasound, 28 825 computed tomography, 14 815 mammography, 12 082 endoscopic, 8 792 magnetic resonance tomography, 3 950 radiology consultation and 1 199 angiographic reports. Word error rates (WER) and report error rates (RER) were calculated for each speaker and modality. Total WER over all material was 18.4% and total RER 93.1%. WER and RER were lowest for mammography dictations (7.7%; 70.3%), and highest for angiography (34.4%; 100%), followed by endoscopy (30.9%; 100%). 3D modalities had higher RER and WER compared to planar x-ray correlating with the complexity of the radiology reports. Live experiments with the ASR prototype showed differences between the users depending on their experience and speech characteristics. In summary, the ASR prototype for Estonian language in radiology domain was the first time successfully applied and assessed in routine clinical practice. Improvements of the ASR prototype performance are planned in the future.