Neural machine translation often suffers from an under-translation problem owing to its limited modeling of the output sequence lengths. In this study, we propose a novel approach to training a Transformer model using length constraints based on length-aware positional encoding (PE). Because length constraints with exact target sentence lengths degrade the translation performance, we add a random perturbation with a uniform distribution within a certain range to the length constraints in the PE during the training. In the inference step, we predicted the output lengths from the input sequences using a length prediction model based on a large-scale pre-trained language model. In Japanese-to-English and English-to-Japanese translation, experimental results show that the proposed perturbation injection improves the robustness of the length prediction errors, particularly within a certain range.
CITATION STYLE
Oka, Y., Sudoh, K., & Nakamura, S. (2021). Length-constrained Neural Machine Translation using Length Prediction and Perturbation into Length-aware Positional Encoding. Journal of Natural Language Processing, 28(3), 778–801. https://doi.org/10.5715/jnlp.28.778
Mendeley helps you to discover research relevant for your work.