A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl

Abstract

Common Crawl is the largest freely available collection of web crawl data and one of the most important sources of pre-training data for large language models (LLMs). It is used so frequently and makes up such large proportions of the overall pre-training data in many cases that it arguably has become a foundational building block for LLM development, and subsequently generative AI products built on top of LLMs. Despite its pivotal role, Common Crawl itself is not widely understood, nor is there much reflection evident among LLM builders about the implications of using Common Crawl's data. This paper discusses what Common Crawl's popularity for LLM development means for fairness, accountability, and transparency in generative AI by highlighting the organization's values and practices, as well as how it views its own role within the AI ecosystem. Our qualitative analysis is based on in-depth interviews with Common Crawl staffers and relevant online documents. After discussing Common Crawl's role in generative AI and how LLM builders have typically used its data for pre-training LLMs, we review Common Crawl's self-defined values and priorities and highlight the limitations and biases of its crawling process. We find that Common Crawl's popularity has contributed to making generative AI more transparent to scrutiny in many ways, and that it has enabled more LLM research and development to take place beyond well-resourced leading AI companies. At the same time, many LLM builders have used Common Crawl as a source for training data in ways that are problematic: for instance, with lack of care and transparency for how Common Crawl's massive crawl data was filtered for harmful content before the pre-training, often by relying on rudimentary automated filtering techniques. We offer recommendations for Common Crawl and LLM builders on how to improve fairness, accountability, and transparency in LLM research and development.