Orchestrating Multi-Agent Systems for Multi-Source Information Retrieval and Question Answering with Large Language Models

Abstract

We present a novel framework for developing robust multi-source questionanswer systems by dynamically integrating Large Language Models with diverse data sources. This framework leverages a multi-agent architecture to coordinate the retrieval and synthesis of information from unstructured documents, like PDFs, and structured databases. Specialized agents, including SQL agents, Retrieval-Augmented Generation agents, and router agents, dynamically select and execute the most suitable retrieval strategies for each query. To enhance contextual relevance and accuracy, the framework employs adaptive prompt engineering, fine-tuned to the specific requirements of each interaction. We demonstrate the effectiveness of this approach in the domain of Contract Management, where answering complex queries often demands seamless collaboration between structured and unstructured data. The results highlight the framework’s capability to deliver precise, context-aware responses, establishing a scalable solution for multi-domain question-answer applications.