This paper presents a novel metric for evaluating stability of machine translation system. A stable system indicates that it keeps almost the same outputs given the inputs with slight changes. In this paper, we propose a stability metric by exploiting TER metric for evaluating the differences between the two texts. We have built an evaluation data set, and demonstrate that a neural-based method is unstable rather than a statistical-based method, while the former outperforms the latter.
CITATION STYLE
Takahashi, K., Takeno, S., & Yamamoto, K. (2017). On evaluation metrics for output stability of machine translation system. Transactions of the Japanese Society for Artificial Intelligence, 32(5), D-H33_1-D-H33_4. https://doi.org/10.1527/tjsai.D-H33
Mendeley helps you to discover research relevant for your work.