Correlation and agreement between Language ENvironment Analysis (lena™) and manual transcription for Dutch natural language recordings

Abstract

The Language ENvironment Analysis system (lena™) automatically analyzes the natural sound environments of children. Among other things, it estimates the amounts of adult words (awc), child vocalizations (cv), conversational turns (ct), and electronic media (tv) that a child is exposed to. To assess lena’s reliability, we compared it to manual transcription. Specifically, we calculated the correlation and agreement between the lena estimates and manual counts for 48 five-min audio samples. These samples were selected from eight day-long recordings of six Dutch-speaking children (ages 2–5). The correlations were strong for awc, r = . 87, and cv, r = . 77, and comparatively low for ct, r = . 52, and tv, r = . 50. However, the agreement analysis revealed a constant bias in awc counts, and proportional biases for cv and ct (i.e., the bias varied with the values for cv and ct). Agreement for detecting electronic media was poor. Moreover, the limits of agreement were wide for all four metrics. That is, the differences between lena and the manual transcriptions for individual audio samples varied widely around the mean difference. This variation could indicate that lena was affected by differences between the samples that did not equally affect the human transcribers. The disagreements and biases cast doubt on the comparability of lena measurements across families and time, which is crucial for using lena in research. Our sample is too small to conclude within which limits lena’s measurements are comparable, but it seems advisable to be cautious of factors that could systematically bias lena’s performance and thereby create confounds.