Reusing weights in subword-Aware neural language models

Abstract

We propose several ways of reusing subword embeddings and other weights in subwordaware neural language models. The proposed techniques do not benefit a competitive character-Aware model, but some of them improve the performance of syllable-and morpheme-Aware models while showing significant reductions in model sizes. We discover a simple hands-on principle: in a multilayer input embedding model, layers should be tied consecutively bottom-up if reused at output. Our best morpheme-Aware model with properly reused weights beats the competitive word-level model by a large margin across multiple languages and has 20%?87% fewer parameters.