We aimed to identify spin in reports of parallel-group RCTs with statisti- cally nonsignificant results for the pri- mary outcome and to develop a scheme for classification of spin strategies. We focused on trials with statistically non- significantprimaryoutcomesbecausethe interpretation of these results are more likely to be affected by a preconceived notion of effectiveness, resulting in a bi- ased interpretation.9 METHODS Selection of Articles The articles were screened from a rep- resentative cohort of articles of RCTs indexed in PubMed. The search strat- egy and eligibility criteria for this co- hort have been described elsewhere.11 Randomized controlled trials were de- fined as prospective studies assessing health care interventions in human par- ticipants randomly allocated to study groups. Reports of cost-effectiveness studies, reports of diagnostic test ac- curacy, and non���English-language re- ports were excluded. In brief, the Cochrane Highly Sensi- tive Search Strategy,12 performed in PubMed to identify primary reports of RCTs published in December 2006 and indexed in PubMed by March 22, 2007, yielded 1735 PubMed citations. After reading the titles and abstracts of re- trieved citations, reports of obviously noneligibletrialswereexcluded,andthe full-text article and any online appendi- ces were obtained and evaluated for 879 selected citations. Of these, 263 cita- tionswereexcludedafterthefulltextwas read the remaining 616 were included in this representative sample of RCTs. Fromthissample,weselectedparallel- group RCTs with clearly identified pri- mary outcomes. We excluded equiva- lence or noninferiority trials, crossover trials, cluster trials, factorial and split- body designs, trials with more than 2 groups, and phase 2 trials. Primary out- comes were those explicitly reported as suchinthepublishedarticle.Ifnonewas explicitly reported, we considered the outcomes stated in the sample size esti- mation if outcomes were not stated in the sample size estimation, we took the outcomes in the primary study objec- tives, if available. If no primary out- come was clearly identified (ie, explic- itly specified in the article, in a sample size calculation, or in the primary study objectives), the article was excluded. One reviewer (I.B.) screened the full- text articles and determined results for all primary outcomes according to sta- tistical significance: results statisti- cally signficant (ie, P .05), results that did not reach statistical significance (ie, P .05), or unclear results. We in- cluded only trials with nonsignificant results (ie, P .05) for all primary out- comes. When no formal statistical analyses were reported for the pri- mary outcomes, we attempted to cal- culate the effect size and confidence in- terval for the primary outcomes, and the article was included if the estimated treatment effect was not statistically sig- nificant. If we could not calculate the effect size using the published data, the article was excluded. Assessment of Selected Articles For each selected article, 2 readers (I.B., S.D.) independently read the title, ab- stract, and Methods, Results, Discus- sion, and Conclusions sections, as well as online appendices referenced in the articles, when available. The reviewers independently appraised the content of the article using a pretested and stan- dardizeddataabstractionform thenthey met to compare results. All discrepan- cies were discussed to obtain consen- sus if needed, the article was discussed with a third reader (D.G.A.). The repro- ducibilitywasmoderate,witha of0.47 (95% confidence interval [CI], 0.27- 0.67) for presence of spin in the ab- stract Conclusions and of 0.64 (95% CI, 0.47-0.82) for spin in the article Con- clusions. General Characteristics of Selected Articles Foreachselectedarticle,werecordedthe funding source (ie, for-profit, non- profit, or both not reported, no fund- ing), 2007 journal impact factor, num- berofcitationsin2008,theexperimental intervention, comparator, sample size, and type of primary outcomes (safety, efficacy, both). Reporting the Primary Outcomes in Abstract and Main Text We checked whether the primary out- comes were clearly identified in the ab- stract. We also recorded the reporting of results for the primary outcomes both in the abstract and in the article (ie, re- porting of estimated effect size with or without precision and reporting of sum- mary statistics [eg, proportion of event, mean] for each group with or without precision). Definition of Spin In the context of a trial with statisti- cally nonsignificant primary out- comes, spin was defined as use of spe- cific reporting strategies, from whatever motive, to highlight that the experi- mental treatment is beneficial, despite a statistically nonsignificant differ- ence for the primary outcome, or to dis- tract the reader from statistically non- significant results. Development of Classification Scheme All of the authors participated in the de- velopment of a classification scheme to standardize the collection of the strate- gies used for spin in the selected re- ports.Forthispurpose,inafirststep,we reviewedtheliteraturepublishedonthis topic.3,6,13-22 We also contacted by e-mail all the members of the Cochrane Statis- tical Method Group and invited them to sendusanyexamplesofpublishedRCTs with spin, in any medical field, and with any publication date. Lastly, we re- viewed a sample of trials with statisti- cally nonsignificant results published in general medical journals with high im- pactfactorsorinspecialistjournals.23 The classification scheme was developed fol- lowingdiscussionandagreementamong the authors. Strategies of Spin Using the developed classification scheme, we searched for spin in each section of the manuscript in our sample, ie, abstract Results abstract Conclu- DISTORTED PRIMARY OUTCOMES PRESENTATION IN RCTS ��2010 American Medical Association. All rights reserved. (Reprinted) JAMA, May 26, 2010���Vol 303, No. 20 2059 at Erasmus MC - Univ of Rotterdam on August 31, 2010 www.jama.com Downloaded from

sions and main-text Results, Discus- sion, and Conclusions (ie, last para- graph of the manuscript when this paragraph summarized the results) sec- tions. We then determined whether authors had used a spin strategy. The strategies of spin considered were (1) a focus on statistically significant re- sults (within-group comparison, sec- ondary outcomes, subgroup analyses, modified population of analyses) (2) interpreting statistically nonsignifi- cant results for the primary outcomes as showing treatment equivalence or comparableeffectiveness and(3)claim- ing or emphasizing the beneficial effect of the treatment despite statistically nonsignificant results. All other spin strategies that could not be classified according to this scheme were system- atically recorded and secondarily classified. Extent of Spin We determined the extent of spin across the whole report, defined as the num- ber of sections with spin in the ab- stract (spin in the Results section only, in the Conclusions section only, or in both sections) and in the main text (spin in one section other than the Con- clusions section, in the Conclusions section only, in 2 sections, or in all 3 sections). The assessment of the ex- tent of spin is exploratory and should not be considered a scoring system. This classification scheme was developed by consensus among the authors for a pragmatic purpose: to be able to cap- ture the diversity of spin in terms of vol- ume (ie, whether spin concerned only a small part or most of the article). Level of Spin in Conclusions We also classified the level of spin in the Conclusions sections of the ab- stract and the main text as follows. High spin was defined as no uncertainty in the framing, no recommendations for further trials, and no acknowledg- ment of the statistically nonsignifi- cant results for the primary outcomes in addition, when the Conclusions sec- tion reported recommendations to use the treatment in clinical practice, we classified this section as having a high level of spin. Moderate spin was de- fined as some uncertainty in the fram- ing or recommendations for further trials but no acknowledgment of the sta- tistically nonsignificant results for the primary outcomes. Low spin was de- fined as uncertainty in the framing and recommendations for further trials or acknowledgment of the statistically nonsignificant results for the primary outcomes. This classification of the level of spin is exploratory and not vali- dated and should not be considered a scoring system. The level of spin was used to explore the heterogeneity of spin in the reporting of conclusions. Statistical Analysis Medians and interquartile ranges for continuous variables and number (%) of articles for categorical variables were calculated. Statistical analyses were per- formed using SAS version 9.1 (SAS In- stitute Inc, Cary, North Carolina). RESULTS General Characteristics of Selected Articles Of the 616 PubMed citations retrieved, 205 reports of parallel-group RCTs were identified. Among these reports, we identified and appraised 72 reports with statistically nonsignificant results for the primary outcomes (FIGURE). Characteristics of the included reports are presented in TABLE 1. Most reports evaluated efficacy (n=63 [87.5% 95% CI, 77.6%-94.1%]), and half evaluated pharmacologicaltreatments.Thefund- ing source was for-profit (only or with a nonprofit source) in one-third of the reportsandwasnotstatedin27(37.5%). Reporting of Primary Outcomes in Abstract and Main Text Primary outcomes were clearly identi- fied in 44 of the 72 report abstracts (61.1% 95% CI, 48.9%-72.4%). In 3 ab- stracts (4.2% 95% CI, 0.9%-11.7%), a secondary outcome was reported as being the primary outcome. Only 9 ab- stracts (12.5% 95% CI, 5.9%-22.4%) reported the effect size and 95% con- fidence interval, and 28 (38.9% 95% CI, 27.6%-51.1%) did not report any numerical results for primary out- comes. In only 16 articles (22.2% 95% CI, 13.3%-33.6%) did the main text de- scribe the effect size and its precision for primary outcomes in 21 (29.2% 95% CI, 19.0%-41.1%), the main text reported only summary statistics for each group, without precision. Spin Strategies The strategies of spin in each article sec- tion are shown in TABLE 2. The title was reported with spin in 13 of the 72 ar- ticles (18.0% 95% CI, 10.0%-28.9%). Spin was identified in 27 (37.5% 95% CI, 26.4%-49.7%) and 42 (58.3% 95% CI, 46.1%-69.8%) of the abstract Re- sults and Conclusions sections, respec- tively. We identified spin in 21 (29.2% 95% CI, 19.0%-41.1%), 31 (43.1% 95% CI, 31.4%-55.3%), and 36 (50.0% 95% CI, 38.0%-62.0%) of the main-text Re- Figure. Study Selection 133 Excluded 124 Statistically significant results (P .05) 9 Unclear results 263 Excluded based on review of full text and online appendices 856 Excluded based on review of titles and abstracts 411 Excluded 122 No primary outcome identified 100 Crossover trial 93 Multiple-group trial 26 Pilot study 16 Split-body design trial 13 Cluster trial 10 Brief communication/letter 10 Factorial trial 10 Noninferiority or equivalence trial 5 Phase 1 trial 3 Phase 2 trial 1 Sequential trial 2 Other 72 RCTs with statistically nonsignificant results for all primary outcomes identified and included in analysis 205 Parallel-group RCTs with clearly identified primary outcomes included 616 Included in sample 879 Identified for further review 1735 Potentially relevant trials identified in PubMed search RCT indicates randomized controlled trial. DISTORTED PRIMARY OUTCOMES PRESENTATION IN RCTS 2060 JAMA, May 26, 2010���Vol 303, No. 20 (Reprinted) ��2010 American Medical Association. All rights reserved. at Erasmus MC - Univ of Rotterdam on August 31, 2010 www.jama.com Downloaded from