Digitally assessing text comprehension in grades 3-4: Test development and validation

Abstract

A prerequisite for child reading support at school is adequate assessment. Embedding (repeated) assessment into daily teaching routine is often challenging for teachers in terms of time and organization. The use of digital tools can help teachers in the assessment process (in preparation, evaluation, documentation, etc.). A digital assessment tool (Graz Reading Comprehension test: GraLeV), focusing on assessing reading comprehension skills in Grades 3 and 4 is currently being developed in Austria. This reading assessment covers reading comprehension at the word, sentence, and text level. Text level is assessed via two subtests (Subtest I: presentation of nonsense-stories and corresponding questions, and Subtest II: maze selection). The other levels consist of one subtest each. This paper focusses on the subtests at text level. More specifically, the paper reports the results of two studies. Study 1 describes the development phases and the first piloting of these two subtests (data collection: 10/2019-12/2019). Testing 273 students with preliminary versions of the subtests (Subtest I: 30 items, Subtest II: 60 items) produced information on (a) item difficulty, (b) item discriminatory power, and (c) time limits for future speed testing. Items not meeting the required quality criteria were excluded. The final version of Subtest I consists of 16 questions referring to eight different, short, nonsense-texts. Its testing time (without instructions) is three minutes. The final version of the Subtest II consists of 2 texts each with 15 maze selections (30 items) and testing time is 100 seconds. The internal consistency is found to be good for Subtest I (α=.87) and Subtest II (α=.78 to .80). Study 2 reports on testing for validity and retest-reliability (data collection: 09/2020-11/2020). Student scores in another reading comprehension test, together with teacher assessments of reading comprehension, were used to assess congruent validity. Divergent validity was assessed using teacher assessments of mathematical and socio-emotional skills. As expected, the correlations with the congruent measures were higher than those with the divergent measures. A subsample was tested twice with the GraLeV. Retest-reliability was acceptable for Subtest II. However, the scores obtained at time 2 were higher compared to those at time 1 in both subtests. This is probably the result of increased student familiarity with the digital device and the digital test environment at time 2. The results are discussed in the light of teachers’ needs for standardized digital assessments in order to facilitate the tailoring of student reading support.