Continual Learning in Multilingual Sign Language Translation

Abstract

The field of sign language translation (SLT) is still in its infancy, as evidenced by the low translation quality, even when using deep learning approaches. Probably because of this, many common approaches in other machine learning fields have not been explored in sign language. Here, we focus on continual learning for multilingual SLT. We experiment with three continual learning methods and compare them to four more naive baseline and fine-tuning approaches. We work with four sign languages (ASL, BSL, CSL and DGS) and three spoken languages (Chinese, English and German). Our results show that incremental fine-tuning is the best performing approach both in terms of translation quality and transfer capabilities, and that continual learning approaches are not yet fully competitive given the current SOTA in SLT.