"I want to believe!" On the importance, uses and limits of inter-coder reliability tests in Content Analysis

Abstract

Introduction: Content analysis has been normatively organized by the following principles: validity, replicability, and reliability. This paper points out that empirical studies in Brazil and abroad have been ignoring such principles, especially the latter (reliability). From this vantage point, we offer a theoretical and practical contribution to content analysis. Methods: Regarding the practical contribution, this papers details check-lists of procedures for conducting intercoder reliability tests for different research conditions (i.e. for one or more coders). Results: Concerning the theoretical contribution, it departs from an epistemological critical assessment of the advantages and limitations embedded in the most common uses of this kind of test. On the limit, such uses might put the scientific reliability of the published results at risk. Discussion: In order to avoid such risk, we argue that empirical studies might claim the presumption of reliability when (a) they offer plain conditions for their replicability; and (b) when they offer a reliability test that might be regarded as significantly non-random. We conclude our paper by pointing out that, in the high ranked journals, prevails the importance of (b) under (a), which implies that even in the elite of scientific production prevails a low demanding comprehension of reliability.