NeurReview: A Neural Architecture Based Conformity Prediction of Peer Reviews

Abstract

Peer review is at the heart of scholarly communications and the foundation of scientific evaluation. However, peer review's effectiveness is continuously challenged due to biased and inconsistent peer reviews. Consequently, ensuring the quality of peer reviews is a time-critical problem. In this paper, we investigate the conformity between reviews and meta-reviews. To predict the review conformity and identify the effective features to distinguish the misaligned reviews, we propose NeurReview, which models the review process from the review structure and interactions with authors and other reviewers. Two evaluation datasets are constructed from the ICLR open reviews. The evaluation results verified the efficacy of our proposed model. In addition, we found that the divergence with other reviews and responses, the consistency of sentiment polarity with the recommendation score, etc., are beneficial features for identifying low-conformity reviews, which can assist meta-reviewers in making final decisions.