Reliability as Lindley Information

Abstract

This paper introduces a definition of reliability based on Lindley information, which is the mutual information between an observed measure and latent attribute. This definition reduces to the traditional definition of reliability in the case of normal variables, but can be applied to any joint distribution of observed and latent variables. Importantly, unlike traditional definitions of reliability, this formulation of reliability applies to vector- or matrix-valued estimates and summaries of responses, and therefore generalizes reliability to sets of scores and estimates in addition to individual scores and estimates. This formulation also leads to new bounds for reliability, as well as newly reported relationships between reliability and the traditional Fisher information function familiar in item response theory literature. This form of reliability can be estimated using formulae, or methods used in Bayesian inference such as Markov Chain Monte Carlo (MCMC) depending on the case. Examples based on well-studied datasets are provided, as well as applications to drift-diffusion modeling and randomly-varying intraindividual covariance structures.