End-to-End Speech Recognition with Deep Fusion: Leveraging External Language Models for Low-Resource Scenarios

Abstract

With the rapid development of Automatic Speech Recognition (ASR) technology, end-to-end speech recognition systems have gained significant attention due to their ability to directly convert raw speech signals into text. However, such systems heavily rely on large amounts of labeled speech data, which severely limits model training performance and generalization, especially in low-resource language environments. To address this issue, this paper proposes an end-to-end speech recognition approach based on deep fusion, which tightly integrates an external language model (LM) with the end-to-end model during the training phase, effectively compensating for the lack of linguistic prior knowledge. Unlike traditional shallow fusion methods, deep fusion enables the model and the external LM to share representations and jointly optimize during training, thereby enhancing recognition performance under low-resource conditions. Experiments conducted on the Common Voice dataset show that, in a 10 h extremely low-resource scenario, the deep fusion method reduces the character error rate (CER) from 51.1% to 17.65%. In a 100 h scenario, it achieves a relative reduction of approximately 2.8%. Furthermore, ablation studies on model layers demonstrate that even with a reduced number of encoder and decoder layers to decrease model complexity, deep fusion continues to effectively leverage external linguistic priors, significantly improving performance in low-resource speech recognition tasks.