Intelligent Site Reliability Engineering: A Multi-agent LLM Framework for Automated Incident Analysis and Root Cause Determination

Abstract

Modern distributed systems generate thousands of daily alerts that overwhelm Site Reliability Engineering (SRE) teams, causing alert fatigue, delayed incident response, and operational knowledge fragmentation. Traditional monitoring approaches using threshold-based alerting and manual correlation achieve only 45-65% accuracy in root cause identification and cannot scale with cloud-native microservices complexity. This paper presents an Intelligent SRE Agent employing a novel multi-agent Large Language Model (LLM) framework for automated incident analysis, root cause determination, and diagnostic reporting. The system implements a hierarchical architecture where a Master LLM Agent coordinates five domain-specialized agents via Model Context Protocol (MCP): Logs Agent for error pattern recognition, Metrics Agent for time-series anomaly detection, Traces Agent for distributed request flow analysis, Infrastructure Agent for platform resource assessment, and ML Services Agent for advanced pattern recognition. This multi-agent approach addresses single-LLM limitations by providing specialized domain expertise across heterogeneous operational data sources. We conducted comprehensive evaluation using dual testbeds: a controlled 7-microservice IoT sensor network with 150 systematically injected failure scenarios over 8 weeks, and an extended 50+ microservice production-replica environment with 200 scenarios. Chaos engineering tools including Toxiproxy, Chaos Monkey, and Litmus Chaos generated realistic failure modes encompassing cascading failures, resource exhaustion, configuration drift, and performance degradations. The Intelligent SRE Agent achieved 92.1% root cause identification accuracy compared to 67.3% manual baseline, reduced mean time to diagnosis by 82% (from 47.2 to 8.5 minutes), and demonstrated 96.3% precision in historical incident matching. Performance remained robust at scale, maintaining 89.4% accuracy in extended validation while handling complex enterprise-grade failures including multi-zone network partitions and cascading service dependencies.