pygetpapers: a Python library for automated retrieval of scientific literature

Abstract

pygetpapers has been developed to allow searching of the scientific literature in repositories with a range of textual queries and metadata. It downloads content using APIs in an automated fashion and is designed to be extensible to the growing number of Open Access repositories. Statement of Need An increasing amount of research, particularly in medicine and applied science, is now based on meta-analysis and systematic review of the existing literature ("Systematic Reviews," 2022). In such reviews, scientists frequently download thousands of articles and analyze them by Natural Language Processing (NLP) through Text and Data Mining (TDM) or Content Mining. A common approach is to search bibliographic resources with keywords, download the hits, scan them manually, and reject papers that do not fit the criteria for the meta-analysis. The typical text-based searches on sites are broad, with many false positives and often only based on abstracts. We know of cases where systematic reviewers downloaded 30,000 articles and eventually used 30. Retrieval is often done by crawling/scraping sites, such as journals but is easier and faster when these articles are in Open Access repositories such as arXiv, EuropePMC, bioRxiv, medRxiv. But each repository has its own API and functionality, which makes it hard for individuals to (a) access, (b) set flags, and (c) use generic queries. In 2015, we reviewed tools for scraping websites and decided that none met our needs and so developed getpapers (Smith-Unna, 2021), with the key advance of integrating a query submission with bulk fulltext-download of all the hits. pygetpapers getpapers was written in NodeJS and has now been completely rewritten in Python3 (pygetpapers) for easier distribution and integration. Typical use of getpapers is shown in a recent paper (Wind et al., 2021) where the authors "analyzed key term frequency within 20,000 representatives [Antimicrobial Resistance] articles". An important aspect is to provide a simple cross-platform approach for scientists who may find tools like curl too complex and want a one-line command to combine the search, download, and analysis into a single: "please give me the results". We've tested this on many interns who learn pygetpapers in minutes. It was also easy to wrap it into a tkinter graphical user interface (GUI) (Lundh, 1999). The architecture of the results is simple and natural, based on full-text files in the normal filesystem. The result of pygetpapers is interfaced using a "main" or "controller" JSON file (for eg. eupmc_results.json), which allows corpus to be reused/added to. This allows maximum flexibility of re-use and some projects have large amounts of derived data in these directories. Garg et al. (2022). pygetpapers: a Python library for automated retrieval of scientific literature. Journal of Open Source Software, 7 (75), 4451. https://doi.org/10.21105/joss.04451.