Questionable science and reproducibility in electrical brain stimulation research

Abstract

Electrical brain stimulation (EBS) is a trendy new technique used to change brain function and treat neurological, psychiatric and psychological disorders. We were curious whether the published literature, which is dominated by positive results, reflects the experience of researchers using EBS. Specifically, we wanted to know whether researchers are able to reproduce published EBS effects and whether they engage in, but fail to report, questionable research practices. We invited 976 researchers to complete an online survey. We also audited 100 randomly-selected published EBS papers. A total of 154 researchers completed the survey. Survey respondents had a median of 3 [1 to 6, IQR] published EBS papers (1180 total) and 2 [1 to 3] unpublished ones (380 total). With anodal and cathodal EBS, the two most widely used techniques, 45-50% of researchers reported being able to routinely reproduce published results. When asked about how study sample size was determined, 69% of respondents reported using the sample size of published studies, while 61% had used power calculations, and 32% had based their decision on pilot data. In contrast, our audit found only 6 papers where power calculations were used and a single paper in which pilot data was used. When asked about questionable research practices, survey respondents were aware of other researchers who selectively reported study outcomes (41%) and experimental conditions (36%), adjusted statistical analysis to optimise results (43%), and engaged in other shady practices (20%). Fewer respondents admitted to engaging in these practices themselves, although 25% admitted to adjusting statistical analysis to optimize results. There was strong agreement that such practices should be reported in research papers; however, our audit found only two such admissions. The present survey confirms that questionable research practices and poor reproducibility are present in EBS studies. The belief that EBS is effective needs to be replaced by a more rigorous approach so that reproducible brain stimulation methods can be devised and applied.