An Agentic AI System for Context-Grounded and Deterministic Failure Triage in Data Pipelines

Abstract

Operational failures in modern data pipelines often require time-consuming manual inspection of large and unstructured logs across heterogeneous systems. This paper presents an agentic, event-driven decision-support framework that transforms raw pipeline failure telemetry into structured incident summaries containing representative evidence, root-cause explanations, and remediation guidance for human operators. The framework is implemented as a set of loosely coupled services connected through a publish-subscribe broker, including telemetry ingestion, anomaly detection, context-grounded root cause analysis (RCA), and notification delivery. A key design feature is deterministic evidence selection prior to automated reasoning, where telemetry signals are aggregated by execution context, noise is suppressed, and a representative failure anchor is selected to improve downstream inference reliability. The RCA component constructs grounded reasoning contexts using execution artifacts such as ETL scripts and schema representations while enforcing strict output constraints and conservative fallback behavior. The system is evaluated using offline replay of 100 AWS Glue ETL failure instances with human-labeled ground truth. Results show 96% top-1 action accuracy and 97% top-3 hit rate within the evaluated AWS Glue offline replay setting. No unsupported infrastructure dependencies, fabricated execution contexts, or unsafe remediation recommendations were observed under the constrained evaluation protocol. An ablation study removing deterministic grounding and representative evidence selection showed that the RCA agent retained broad diagnostic utility but produced less specific remediation guidance and a small number of unsupported contextual assumptions. Evidence selection compresses an average of 16.18 telemetry lines into a single representative failure signal while maintaining perfect anchor precision. The framework demonstrates low-latency near-real-time diagnostic performance within the evaluated offline replay environment with a median alert-to-notification latency of 8 seconds. These results suggest that combining deterministic preprocessing with context-grounded reasoning can improve diagnostic consistency and constrained operational safety characteristics within offline data pipeline failure triage environments.