A controlled experiment on the process used by developers during internet-scale code search

Abstract

It has become common practice for developers to search the Web for source code. In this paper, we report on our analysis of a laboratory experiment with 24 subjects. They were given a programming scenario and asked to find source code using five different search engines. The scenarios varied in terms of size of search target (block or subsystem) and usage intention (as-is reuse or reference example). Every subject used five search engines (Google, Koders, Krugle, and Google Code Search, and SourceForge). We looked at how these factors influenced three phases of the search process: query formulation, query revision, and judging relevance. One consistent trend was searching for reference examples required more effort, as measured by average number of terms per query, average number of queries, clickthrough rate, and time spent. This additional effort paid off in a higher rate of precision for the first ten results.