Applications of automatic evaluation methods to measuring a capability of speech translation system

Abstract

The main goal of this paper is to propose automatic schemes for the translation paired comparison method. This method was proposed to precisely evaluate a speech translation system's capability. Furthermore, the method gives an objective evaluation result, i.e., a score of the Test of English for International Communication (TOEIC). The TOEIC score is used as a measure of one's speech translation capability. However, this method requires tremendous evaluation costs. Accordingly, automatization of this method is an important subject for study. In the proposed method, currently available automatic evaluation methods are applied to automate the translation paired comparison method. In the experiments, several automatic evaluation methods (BLEU, NIST, DP-based method) are applied. The experimental results of these automatic measures show a good correlation with evaluation results of the translation paired comparison method.