COMPUTER ANALYSIS OF THE TOEFL TEST OF WRITTEN ENGLISH

Abstract

This project had three main objectives: (1) to establish a database of essays written by different language groups on a variety of topics for the Test of Written English (TWE) that can be used in future research; (2) to summarize, analyze, and compare the linguistic properties of those essays; and (3) to determine how the TWE performance of language groups relates to essay styles. As part of the first objective of this project we created a database of 1,737 essays, a data matrix of essay variables, files containing sorted phrases and vocabularies of different language groups, and files of the common and unique vocabulary items for each pair of language groups. The essay sample consisted of TWE essays from five language groups, including Arabic, Chinese, English (including native-English and nonnative-English), and Spanish speakers. Essays from English-speaking students in the United States were collected and scored to provide a baseline with which to compare essays by students who speak English as a second language (ESL). This report includes the analysis of 106 variables for each essay along with summary analyses, such as correlation, analyses of variance, discriminative analysis, and factor analysis. It also presents information on the accuracy and cost of data entry and on the accuracy of the text analysis programs, as well as extensive data on vocabulary. A program that assesses content was developed for this project. Results show that topic differences affected some essay variables, but these effects were generally felt equally by the different language groups. Factor analysis replicated work showing that major distinguishing features of academic writing include a nominal style, passive constructions, and complexity of sentence structure. Discriminant analysis suggest three features that might distinguish the performance of different language groups — directness, expressiveness, and academic stance. Nevertheless, a linguistic analysis of the accuracy of underlying text analysis programs shows that for some word classes implicated in defining “academic” or “expressive” styles, cautions are needed in interpreting program outputs. Two variables that can be measured unambiguously by computer — number of words and the average length of words — taken together are quite predictive of TWE essay scores of non-English speakers (multiple R >.8).