RAG Without the Lag: Enabling "what-If" Analysis for Retrieval-Augmented Generation Pipelines

Abstract

Retrieval-augmented generation (RAG) pipelines have become the de-facto approach for building AI assistants with external knowledge. Given a user query, RAG pipelines retrieve (R) information from external sources, before invoking a Large Language Model (LLM), augmented (A) with this information, to generate (G) responses. However, developing effective RAG pipelines is challenging because retrieval and generation components - often chained in varying orders - are intertwined, making it hard to identify which component(s) cause errors in the output. Developers often need to answer "what-if"questions - e.g., what if chunk sizes were larger or retrieval used embeddings versus keywords - but such experimentation requires hours of re-processing. We present raggy, a developer tool that enables rapid "what-if"analysis by combining a Python library of composable RAG primitives with an interactive debugging interface. We contribute the design and implementation of raggy, insights into expert debugging patterns through a qualitative study with 12 engineers, and design implications for RAG tools.