Sampling the web: The development of a custom search tool for research

Abstract

Research designed to study the Internet is beset with challenges. One of these challenges involves obtaining samples of Web pages. Methodologies used in previous studies may be categorized into random, purposeful, and purposeful random types of sampling. This paper contains an outline of these methodologies and information about the development of a custom sampling tool that may be used to obtain purposeful random samples of Web page links. The custom search application called Web Sampler works through the Google Web APIs service to collect a random sample of pages from search results returned from the Google index. Web Sampler is inexpensive to develop and may be easily customized for specialized search needs required by researchers who are investigating Web page content.