Automated Research Review Support Using Machine Learning, Large Language Models, and Natural Language Processing

Abstract

Research expands the boundaries of a subject, economy, and civilization. Peerreview is at the heart of research and is understandably an expensive process. This work,with human-in-the-loop, aims to support the research community in multiple ways. Itpredicts quality, and acceptance, and recommends reviewers. It helps the authors andeditors to evaluate research work using machine learning models developed based on adataset comprising 18,000+ research papers, some of which are from highly acclaimed,top conferences in Artificial Intelligence such as NeurIPS and ICLR, their reviews, aspectscores, and accept/reject decisions. Using machine learning algorithms such as SupportVector Machines, Deep Learning Recurrent Neural Network architectures such as LSTM, awide variety of pre-trained word vectors using Word2Vec, GloVe, FastText, transformerarchitecture-based BERT, DistilBERT, Google’s Large Language Model (LLM), PaLM 2, andTF-IDF vectorizer, a comprehensive system is built. For the system to be readily usable andto facilitate future enhancements, a frontend, a Flask server in the cloud, and a NOSQLdatabase at the backend are implemented, making it a complete system. The work is novelin using a unique blend of tools and techniques to address most aspects of building asystem to support the peer review process. The experiments result in a 86% test accuracyon acceptance prediction using DistilBERT. Results from other models are comparable, withPaLM-based LLM embeddings achieving 84% accuracy.