Searching relevant papers for software engineering secondary studies: Semantic Scholar coverage and identification role

Abstract

Searching relevant papers is a fundamental task for the elaboration of secondary studies. This task is known to be tedious and time-consuming when it is made manually, especially with the presence of several academic repositories. Recently, Semantic Scholar has emerged as a new artificial intelligence-based search engine enabling a set of valuable features. The present study investigates the role of Semantic Scholar in retrieving relevant papers for performing secondary studies in software engineering. For this sake, an examination is performed to check the ability of Semantic Scholar to locate included papers in recent and well-established secondary studies. Afterwards, a hybrid and automatic search strategy is introduced making use of Semantic Scholar as a sole search engine and it incorporates: automatic search, snowballing, and use of Computer Science Ontology (CSO) and Software Engineering Body of Knowledge (SWEBOK) for refining queries. The proposed strategy is validated by replicating the search of high-quality secondary studies in the software engineering field. To guarantee objectivity, a systematic search is conducted of recent secondary studies published in the field since 2015. For the coverage test, Semantic Scholar is examined to locate primary papers of selected secondary studies and identify missing venues. The proposed search strategy is used to check the ability to retrieve primary papers of each secondary study. The systematic search yielded 20 high-quality secondary studies with 1337 distinct primary papers. The coverage test revealed that Semantic Scholar covers 98.88% of the papers. The proposed search strategy enabled the full replication of 13 studies and more than 90% for the 7 remaining studies.