Assessing transcription agreement: Methodological aspects

  • Cucchiarini C
  • 23

    Readers

    Mendeley users who have this article in their library.
  • 55

    Citations

    Citations of this article.

Abstract

In recent years it has become common practice among speech researchers to report transcription agreement coefficients, when describing findings based on phonetic transcription. Although such coefficients are intended to give an indication of the degree of transcription accuracy, in reality it is not clear to what extent high agreement coefficients do indeed guarantee great transcription accuracy. In this paper it is argued that the most commonly used agreement coefficient, percentage agreement, has three major disadvantages when applied to phonetic transcription: (a) it is based on the assumption that agreement between transcription symbols is all-or-none, (b) it is strongly influenced by chance agreement, and (c) it gives no account of the criteria used for transcription alignment. After a detailed discussion of these drawbacks, an alternative approach to calculating transcription (dis)agreement is proposed. In this approach experimentally derived feature matrices are used as input to a program that aligns transcription pairs automatically and at the same time calculates a (dis)agreement measure for each transcription pair, the average distance. It is argued that this metric is more adequate to express the degree of (dis)similarity between transcriptions than the more usual percentage agreement. The results of an evaluation experiment corroborate this view.

Author-supplied keywords

  • Phonetic transcription
  • Transcription agreement
  • Transcription assessment

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document

Authors

  • Catia Cucchiarini

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free