Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAFA: A multi-agent framework for annotation

Mahmood Hegazy, Aaron Rodrigues, Azzam Naeem

TL;DR

The paper addresses the challenge of mapping user utterances to relevant FAQs in banking by proposing MAFA, a multi-agent framework that combines four specialized ranker agents with a dedicated judge agent for final reranking. It leverages Attentive Reasoning Queries (ARQs) with structured JSON prompts and a diverse few-shot strategy to improve coverage and reduce reasoning errors. Empirical results on an internal banking dataset and public benchmarks (LCQMC and FiQA) show substantial gains over BM25 and single-agent baselines, including notable improvements in Top-1, Top-5, and MRR, with strong generalization across domains and languages. The work emphasizes production-readiness through parallel execution, interpretability of reasoning, and robust performance without extensive domain-specific fine-tuning, while outlining future directions for dynamic agent selection and broader societal considerations.

Abstract

Modern consumer banking applications require accurate and efficient retrieval of information in response to user queries. Mapping user utterances to the most relevant Frequently Asked Questions (FAQs) is a crucial component of these systems. Traditional approaches often rely on a single model or technique, which may not capture the nuances of diverse user inquiries. In this paper, we introduce a multi-agent framework for FAQ annotation that combines multiple specialized agents with different approaches and a judge agent that reranks candidates to produce optimal results. Our agents utilize a structured reasoning approach inspired by Attentive Reasoning Queries (ARQs), which guides them through systematic reasoning steps using targeted, task-specific JSON queries. Our framework features a few-shot example strategy, where each agent receives different few-shots, enhancing ensemble diversity and coverage of the query space. We evaluate our framework on a real-world major bank dataset as well as public benchmark datasets (LCQMC and FiQA), demonstrating significant improvements over single-agent approaches across multiple metrics, including a 14% increase in Top-1 accuracy, an 18% increase in Top-5 accuracy, and a 12% improvement in Mean Reciprocal Rank on our dataset, and similar gains on public benchmarks when compared with traditional and single-agent annotation techniques. Our framework is particularly effective at handling ambiguous queries, making it well-suited for deployment in production banking applications while showing strong generalization capabilities across different domains and languages.

MAFA: A multi-agent framework for annotation

TL;DR

The paper addresses the challenge of mapping user utterances to relevant FAQs in banking by proposing MAFA, a multi-agent framework that combines four specialized ranker agents with a dedicated judge agent for final reranking. It leverages Attentive Reasoning Queries (ARQs) with structured JSON prompts and a diverse few-shot strategy to improve coverage and reduce reasoning errors. Empirical results on an internal banking dataset and public benchmarks (LCQMC and FiQA) show substantial gains over BM25 and single-agent baselines, including notable improvements in Top-1, Top-5, and MRR, with strong generalization across domains and languages. The work emphasizes production-readiness through parallel execution, interpretability of reasoning, and robust performance without extensive domain-specific fine-tuning, while outlining future directions for dynamic agent selection and broader societal considerations.

Abstract

Modern consumer banking applications require accurate and efficient retrieval of information in response to user queries. Mapping user utterances to the most relevant Frequently Asked Questions (FAQs) is a crucial component of these systems. Traditional approaches often rely on a single model or technique, which may not capture the nuances of diverse user inquiries. In this paper, we introduce a multi-agent framework for FAQ annotation that combines multiple specialized agents with different approaches and a judge agent that reranks candidates to produce optimal results. Our agents utilize a structured reasoning approach inspired by Attentive Reasoning Queries (ARQs), which guides them through systematic reasoning steps using targeted, task-specific JSON queries. Our framework features a few-shot example strategy, where each agent receives different few-shots, enhancing ensemble diversity and coverage of the query space. We evaluate our framework on a real-world major bank dataset as well as public benchmark datasets (LCQMC and FiQA), demonstrating significant improvements over single-agent approaches across multiple metrics, including a 14% increase in Top-1 accuracy, an 18% increase in Top-5 accuracy, and a 12% improvement in Mean Reciprocal Rank on our dataset, and similar gains on public benchmarks when compared with traditional and single-agent annotation techniques. Our framework is particularly effective at handling ambiguous queries, making it well-suited for deployment in production banking applications while showing strong generalization capabilities across different domains and languages.
Paper Structure (45 sections, 1 figure, 8 tables, 1 algorithm)

This paper contains 45 sections, 1 figure, 8 tables, 1 algorithm.

Table of Contents

  1. Related work
  2. FAQ retrieval systems
  3. Multi-agent systems
  4. Retrieval-augmented generation (RAG)
  5. Judging and reranking in information retrieval
  6. Structured reasoning for large language models
  7. Problem formulation
  8. System architecture
  9. Query planning agent
  10. Ranker agent network
  11. Structured agent prompting
  12. Judge agent
  13. Implementation
  14. Model selection
  15. Framework implementation
  16. ...and 30 more sections

Figures (1)

  • Figure 1: Architecture of MAFA. Each agent generates candidate FAQ annotations that are re-ranked by the judge agent. All agents have access to the FAQ database.