An audit of the statistical validity of conclusions of clinical superiority in anaesthesia journals

Abstract

Making a statistically valid conclusion of the superiority of a clinical intervention in a clinical trial requires not only a statistically significant P value, but also adequate a priori power and an observed effect size larger than the clinically important value specified in the sample size calculation. We scrutinised the five most highly cited clinical trials reporting one or more conclusions of clinical superiority published in Anesthesiology, the British Journal of Anaesthesia, Anaesthesia, Anesthesia and Analgesia and Anaesthesia and Intensive Care in 2011 or 2012 to determine how many met all three requisite criteria. In the 25 articles, there were a total of 36 unconditional conclusions of the superiority of a clinical intervention. All were supported by a statistically significant P value. However, only 15 (41.6%) met all three requisite statistical criteria to support clinical superiority. The remainder included secondary outcomes without specific reference to their observational nature, and primary outcomes whose observed effect size was smaller than the clinically important value specified in the sample size calculation. These findings indicate that clinicians should closely scrutinise conclusions of clinical superiority in anaesthesia journals. Many will be 'hypothesis-generating observations' without adequate statistical support for a conclusion of clinical superiority in their own right.