Feasibility and desirability of screening search results from Google Search exhaustively for systematic reviews: A cross-case analysis

Abstract

A commonly reported challenge of using Google Search to identify studies for a systematic review is the high number of results retrieved. Thus, ‘stopping rules’ are applied when screening, such as screening only the first 100 results. However, recent evidence shows that Google Search estimates a much higher number of results than the viewable number, raising the possibility of exhaustive screening. This study aimed to provide further evidence on the feasibility of screening search results from Google Search exhaustively, and to assess the desirability of this in terms of identifying studies for a systematic review. We conducted a cross-case analysis of the search results of eight Google Search searches from two systematic reviews. Feasibility of exhaustive screening was ascertained by calculating the viewable number of results. Desirability was ascertained according to: (1) the distribution of studies within the results, irrespective of relevance to a systematic review; (2) the distribution of studies which met the inclusion criteria for the two systematic reviews. The estimated number of results across the eight searches ranged from 342,000 to 72,300,000. The viewable number ranged from 272 to 364. Across the eight searches the distribution of studies was highest in the first 100 results. However, the lowest ranking relevant studies were ranked 227th and 215th for the two systematic reviews. One study per review was identified uniquely from searching Google Search, both within the first 100 results. The findings suggest it is feasible and desirable to screen Google Search results more extensively than commonly reported.