Investigating Acoustic Correlates of Whisper Scoring for L2 Speech Using Forced Alignment with the Italian Component of the ISLE corpus

Abstract

Automatic Speech Recognition (ASR) can be used to analyse L2 speech but researchers cannot be sure that the ASR transcriptions accurately represent learner speech. We aim to confront the ASR outputs with the acoustic analysis of learner speech. Whisper (Radford, 2023) provides transcriptions and probabilities associated to the predicted transcriptions. This paper analyses how global phonetic analyses of learner data can be used to potentially confirm these Whisper probability scores assigned to learner transcriptions. We tested the Italian component of the ISLE corpus with phonetic analyses of 23 learners of English. We compared the levels assigned to these speakers by the corpus experts to the outputs of Whisper’s tiny model. We discuss the phonetic features that may account for these Whisper predictions using acoustic data extracted from forced alignment. We try to correlate the levels assigned to the speakers in the ISLE corpus with the quality of the phonetic realisation, using global vocalic measurements such as the convex hull or Euclidian distances between monophthongs. We show that Levenshtein distance to the reference transcription of the Whisper tiny model (measured using Levenshtein distance to the read text) correlates with the grades assigned by the annotators.