The Reliability of Scores from Affective Instruments

Abstract

In psychometrics, reliability is used to describe the consistency of a measure. Reliability can be understood in the terms when describing its meaning in the instrument development process. The goal of reliability becomes the determination of how much variability in the instrument score is due to “measurement error” and how much is due to variability in “true score” of the respondent. This chapter approaches reliability from the classical test theory (CTT) model. Most importantly in this chapter, however, the focus is on the concept of correlation and how to index it for purpose of understanding the “reliability” or consistency of the instrument under the CTT framework. This chapter discusses a variety of ways that reliability can be understood and how each represent distinct quantifications of consistency. Through several examples, different types of reliability are illustrated along with “acceptable” levels of consistency that should be expected from their measurement of characteristics in the affective domain. One of the central limitations of true score theory is that only one type of error can be addressed at a time. This chapter provides brief introduction to the conceptual framework of Generalizability theory (G-theory), which provides one potential solution to this issue. The chapter concludes with a discussion that links validity and reliability, the two central concepts that frame all the work behind the development of instruments for measuring affective characteristics.