Self-attention Networks for Non-recurrent Handwritten Text Recognition

Abstract

Handwritten text recognition is still an unsolved problem in the field of machine learning. Nevertheless, this technology has improved considerably in the last decade in part thanks to advancements in recurrent neural networks. Unfortunately, due to their sequential nature, recurrent models cannot be effectively parallelised during training. Meanwhile, in natural language processing research, the transformer has recently become the dominant architecture; replacing the recurrent networks that were once popular. These new models are far more efficient to train than their predecessors because their primary building block, the self-attention network, can process sequences entirely non-recurrently. This work demonstrates that self-attention networks can replace the recurrent networks of state-of-the-art handwriting recognition models and achieve competitive error rates, while reducing the time required to train and the number of parameters significantly.