Dynamic Multi-Agent Orchestration and Retrieval for Multi-Source Question-Answer Systems using Large Language Models

TL;DR

The paper addresses the challenge of building accurate, context-aware Q&A systems that integrate information from heterogeneous sources (unstructured documents and structured databases). It proposes a dynamic multi-agent architecture that orchestrates Retrieval-Augmented Generation (RAG), Text-to-SQL, and dynamic prompt engineering, guided by a Router Agent. Implemented in Contrato360 for Contract Management, the framework demonstrates improved data coverage and relevance by routing queries to the most appropriate data source and tailoring prompts in real time. Key contributions include detailed chunking strategies, embedding model choices, vectorstore selection (ChromaDB), and visual-data support via a Graph Agent, yielding a scalable, cross-domain approach to multi-source information retrieval without model retraining.

Abstract

We propose a methodology that combines several advanced techniques in Large Language Model (LLM) retrieval to support the development of robust, multi-source question-answer systems. This methodology is designed to integrate information from diverse data sources, including unstructured documents (PDFs) and structured databases, through a coordinated multi-agent orchestration and dynamic retrieval approach. Our methodology leverages specialized agents-such as SQL agents, Retrieval-Augmented Generation (RAG) agents, and router agents - that dynamically select the most appropriate retrieval strategy based on the nature of each query. To further improve accuracy and contextual relevance, we employ dynamic prompt engineering, which adapts in real time to query-specific contexts. The methodology's effectiveness is demonstrated within the domain of Contract Management, where complex queries often require seamless interaction between unstructured and structured data. Our results indicate that this approach enhances response accuracy and relevance, offering a versatile and scalable framework for developing question-answer systems that can operate across various domains and data sources.

Figures (8)

• Figure 1: Retrieval-Augmented Generation.
• Figure 2: Chunking based on Contract's clauses
• Figure 3: Chunk's metadata
• Figure 4: Agents Architecture.
• Figure 5: Application architecture.