Degrees of freedom & the case study (from qualitative & quantitative methods in evaluation research

Abstract

SCIENCE IS ABLE TO CONTRADICT SOME COMMONSENSE ASSUMPTIONS ONLY BY TRUSTING THE GREAT BULK OF COMMONSENSE KNOWLEDGE. AN EXAMPLE IS GIVEN OF TWO IDENTICAL LINES THAT, BECAUSE OF AN OPTICAL ILLUSION, APPEAR TO BE OF DIFFERENT LENGTHS. BY USING A RULER (QUANTITATIVE MEASUREMENT), ONE CAN REJECT ONE'S INACCURATE VISUAL PERCEPTION (QUALITATIVE COMMONSENSE KNOWING). HOWEVER, TRUSTING THE RULER IMPLIES TRUSTING A LARGER SET OF VISUAL PERCEPTIONS AND ASSUMPTIONS CONCERNING THE RULER ITSELF AND THE MEASUREMENT PROCESS. APPLIED TO EVALUATION RESEARCH, SUCH EPISTEMOLOGY SUGGESTS THAT THE NARRATIVE HISTORY PART OF MOST REPORTS SHOULD BE LEGITIMIZED AND THAT COMMONSENSE PROGRAM CRITIQUES AND EVALUATIONS GENERATED AMONG THE PROGRAM STAFF, PROGRAM CLIENTS, THEIR FAMILIES, AND COMMUNITY OBSERVERS SHOULD BE SYSTEMATICALLY TAPPED. FURTHERMORE, NONQUANTITATIVE METHODS OF COLLECTION AND COMPILING, SUCH AS HIERARCHICALLY ORGANIZED DISCUSSION GROUPS, SHOULD BE ADDED TO QUANTITATIVE PROCEDURES. WHERE SUCH EVALUATIONS ARE CONTRARY TO THE QUANTITATIVE RESULTS, THE RESULTS SHOULD BE REGARDED AS SUSPECT UNTIL THE REASONS FOR THE DISCREPANCY ARE WELL UNDERSTOOD. USE OF SINGLE CASE STUDIES AS THE DOMINANT MODE OF STUDY IN ANTHROPOLOGY, COMPARATIVE POLITICAL SCIENCES, AND COMPARATIVE SOCIOLOGY RAISES SUCH PROBLEMS AS THE INABILITY TO INTERPRET COMPARISONS BETWEEN BUT TWO NATURAL INSTANCES AND THE LACK OF DEGREES OF FREEDOM FOR POSTDICTIONS COMPARED TO PREDICTIONS. SUGGESTIONS CAN BE MADE FOR IMPROVING THE DISCIPLINE BASED ON ANALOGS FROM QUANTITATIVE STUDIES. RESEARCHERS SHOULD KEEP MORE EXPLICIT RECORDS OF ALL THEORIES CONSIDERED IN THE CREATIVE PUZZLE-SOLVING PROCESS. AN ADVERSARY PROCESS SHOULD BE FOSTERED SO THAT OUTSIDE EXPERTS CAN LOOK FOR OTHER IMPLICATIONS OF A THEORY AND FACTS THAT MIGHT CONTRADICT OR SUPPORT IT. ETHNOGRAPHERS COULD USE A SUGGESTED METHOD OF TRIANGULATION THROUGH THE CULTURAL BIAS OF OBSERVERS. FIGURES INDICATE THE MULLER-LYER PERCEPTION ILLUSTRATION AND MULTIPLE-ETHNOGRAPHY SCHEDULES. REFERENCES ARE GIVEN. (AOP)