Robust Dysarthric Speech Recognition with GAN Enhancement and LLM Correction

Abstract

Dysarthric speech recognition faces significant challenges of acoustic variability and data scarcity, and this study proposes a robust system by integrating generative adversarial network enhancement and large language model correction to address these issues effectively. The system employs three key components, including a multimodal recognition core that combines whisper-medium encoder with LoRA-fine-tuned Llama-3.1-8B for end-to-end acoustic-to-semantic mapping, an improved CycleGAN module that generates synthetic dysarthric speech through Inception-ResNet fusion blocks, and an intelligent error correction mechanism using N-best hypothesis reranking with semantic constraints. Experiments on the UA-Speech dataset show that the complete system achieves a 20.61% word error rate representing a 73.9% relative improvement over traditional end-to-end transformer automatic speech recognition. Under very low intelligibility conditions it maintains a 48.69% word error rate demonstrating robust recognition for severe pathological speech. Ablation studies validate each module's effectiveness, providing significant advances for dysarthric patient communication technologies.