No “gold standard” critical appraisal tool for allied health research

Abstract

Data sources Electronic database and internet site searching was performed with no language or time limits. The search terms were broad, and consisted of words such as critical appraisal tools, critical appraisal, critical review form, systematic review form, appraisal of research methodology and research design review. Study selection Critical appraisal tools were included if they were applicable to at least one research design (quantitative and qualitative research), had clear and unambiguous criteria, and could produce a numeric quality score. Critical appraisal instruments that were not published in full, or were not in English, were excluded. Tools that were used for appraisal of diagnostic instruments and clinical guidelines were excluded. Data extraction and synthesis The critical appraisal tools were classified according to the study design for which they were intended. Their items were then classified into one of 12 criteria based on their intent. Commonly occurring items were identified. The empirical basis for construction of the tool, the method by which overall quality of the study was established, psychometric properties of the critical appraisal tools and whether guidelines were provided for their use were also recorded. Results The search identified 193 articles, of which 108 met inclusion criteria. A total of 121 different critical appraisal tools were retrieved. Eighty-seven percent of critical appraisal tools were specific to a research design, with most tools having been developed for experimental studies. There was considerable variability in items contained in the critical appraisal tools. Twelve percent of available tools were developed using specified empirical research. Forty-nine percent of the critical appraisal tools summarised the quality appraisal into a numeric summary score. Few critical appraisal tools had documented evidence of validity of their items, or reliability of use. Guidelines regarding administration of the tools were provided in 43% of cases. Conclusions There was considerable variability in intent, components, construction and psychometric properties of published critical appraisal tools for research reports. There is no “gold standard” critical appraisal tool for any study design, nor is there any widely accepted generic tool that can be applied equally well across study types. No tool was specific to allied health research requirements. Thus, interpretation of critical appraisal of research reports currently needs to be considered in light of the properties and intent of the critical appraisal tool chosen for the task.