Evaluating Tracking Accuracy of an Automatic Reading Tutor

Abstract

In automatic reading tutoring, tracking is the process of automatically following a reader through a given target text. When developing tracking algorithms, a measure of the tracking accuracy – how often a spoken word is aligned to the right target text word position – is needed in order to evaluate performance and compare different algorithms. This paper presents a framework for determining the observed tracking error rate. The proposed framework is used to evaluate three tracking strategies: A) follow the reader to whichever word he/she jumps to in the text, B) follow the reader monotonically from left to right ignoring word skips and regressions (going back to a previous text word), and C) the same as B but allowing isolated word skips. Observed tracking error rate for each of the three tracking strategies is: A: 53%, B: 56%, and C: 47%, on 1883 utterances from 25 children.