Impact of inter-judge agreement on perceptual judgment of nasality

Abstract

Purpose: To investigate the effect of perceptual inter-judge agreement of hypernasality on velopharyngeal (VP) closure prediction. Methods: Two logistic regression models were developed aiming to verify the possibility of predicting the VP closure using the following characteristics: rating of VP closure (adequate, borderline, inadequate), determined by the pressure-flow technique, degree of hypernasality (absent, mild, moderate, severe), and the presence/absence of nasal air emission and nasal rustle determined perceptually by three experienced speech language pathologists. In the first model, 100 speech samples with a moderate agreement rate of hypernasality (kappa coefficient: 0.41) were used. In the second model, 43 speech samples with a perfect agreement among judges were included. The χ2-test was used to compare the models (p≤0.05). Results: In the first model, 65 of the 100 samples were rated in the correct VP closure category, with 42 adequate and 23 inadequate. The borderline VP closure was not predicted. The second model rated 31 of the 43 samples in the correct category, with 21 adequate VP closure, 5 in the borderline VP closure, and 5 inadequate. There was no difference (p=0.526) between the two models. However, the second model showed a higher proportion of accuracy (7%) than the first one, and it has also predicted the borderline VP closure. Conclusion: These results showed the importance of high index of inter-judge agreement when using subjective parameters of speech evaluation, especially when compared to an instrumental evaluation. This suggests the need for strategies for training and calibration of judges in the perceptual judgment to improve the reliability of auditory-perceptual assessment.