RAG-Governor: Unified Governance for Trustworthy and Cost-Efficient Retrieval-Augmented Generation with SLO Guarantees

Abstract

Retrieval-Augmented Generation (RAG) grounds language model outputs in external knowledge to reduce hallucinations, yet current architectures treat retrieval as a one-time preprocessing step that cannot detect errors during generation, adapt to varying source reliability, or optimize costs under latency constraints. We present RAG-Governor, a unified governance layer that reframes RAG as continuous monitoring and adaptation. The system integrates three modules: a validation engine that performs real-time entity extraction and entropy-based uncertainty monitoring to intercept hallucinations during token generation, a source manager that maintains dynamic trust scores through cross-verification feedback loops, and a hierarchical memory system combining knowledge graphs with semantic caching to reduce retrieval costs while ensuring information freshness. Evaluation across four benchmarks spanning factual and temporal question answering demonstrates that RAG-Governor reduces hallucination rates by up to 42% compared to reliability-aware baselines while decreasing retrieval costs by 40-52% through intelligent cache utilization. The system achieves 56% higher throughput than standard RAG and maintains SLO violation rates below 20% under strict latency budgets where existing methods exceed 45%. These results establish continuous governance as a practical paradigm for deploying trustworthy and cost-efficient RAG systems in production environments.