Comparison of methods for evaluating complexity of simplified texts among deaf and hard-of-hearing adults at diferent literacy levels

Abstract

Research has explored using Automatic Text Simplifcation for reading assistance, with prior work identifying benefts and interests from Deaf and Hard-of-Hearing (DHH) adults. While the evaluation of these technologies remains a crucial aspect of research in the area, researchers lack guidance in terms of how to evaluate text complexity with DHH readers. Thus, in this work we conduct methodological research to evaluate metrics identifed from prior work (including reading speed, comprehension questions, and subjective judgements of understandability and readability) in terms of their efectiveness for evaluating texts modifed to be at various complexity levels with DHH adults at diferent literacy levels. Subjective metrics and low-linguistic-complexity comprehension questions distinguished certain text complexity levels with participants with lower literacy. Among participants with higher literacy, only subjective judgements of text readability distinguished certain text complexity levels. For all metrics, participants with higher literacy scored higher or provided more positive subjective judgements overall.