Is Peer Review a Game of Chance?

  • Neff B
  • Olden J
  • 151

    Readers

    Mendeley users who have this article in their library.
  • 39

    Citations

    Citations of this article.

Abstract

Peer review is the standard that journals and granting agencies use to ensure the scientific quality of their publications and funded projects. The peer-review process continues to be criticized, but its actual effectiveness at ensuring quality has yet to be fully investigated.Here we use probability theory to model the peer-review process, focusing on two key components: (1) editors’ prescreening of submitted manuscripts and (2) the number of referees polled. The model shows that the review process can include a strong “lottery” component, independent of editor and referee integrity. Focusing on journal publications, we use a Bayesian approach and citation data from biological journals to show that top journals successfully publish suit- able papers—that is, papers that a large proportion of the scientific community would deem acceptable—by using a prescreening process that involves an editorial board and three referees; even if that process is followed, about a quarter of published papers still may be unsuitable. The element of chance is greater if journals engage only two referees and do no prescreening (or if only one editor prescreens); about half of the papers published in those journals may be unsuitable. Furthermore, authors whose manuscripts were initially rejected can significantly boost their chances of being published by resubmitting their papers to other journals.We make three key recommendations to ensure the integrity of scientific publications in journals: (1) Use an editor or editorial board to prescreen and remove manuscripts of low suitability; (2) use a three-of-three or four-of-four decision rule when deciding

Author-supplied keywords

  • bayesian approach
  • citation
  • impact
  • probability
  • publication bias

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document

Authors

  • Bryan D. Neff

  • Julian D. Olden

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free