Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAFA: A Multi-Agent Framework for Enterprise-Scale Annotation with Configurable Task Adaptation

Mahmood Hegazy, Aaron Rodrigues, Azzam Naeem

TL;DR

MAFA presents a production-deployed, configurable multi-agent framework for enterprise annotation that combines specialized agents, structured reasoning, and a judge-based consensus to achieve high accuracy at scale. By enabling dynamic task adaptation and integrating with human-in-the-loop workflows, it eliminates large backlogs and delivers measurable business benefits in a banking context. Key innovations include a configuration-driven architecture, ARQ-based structured prompts, few-shot diversity, embedding-enhanced retrieval, and a robust consensus mechanism that preserves auditability and compliance. The approach demonstrates strong cross-domain and multilingual performance, practical latency and scalability, and meaningful operational impact, offering a blueprint for enterprise AI deployments facing annotation bottlenecks.

Abstract

We present MAFA (Multi-Agent Framework for Annotation), a production-deployed system that transforms enterprise-scale annotation workflows through configurable multi-agent collaboration. Addressing the critical challenge of annotation backlogs in financial services, where millions of customer utterances require accurate categorization, MAFA combines specialized agents with structured reasoning and a judge-based consensus mechanism. Our framework uniquely supports dynamic task adaptation, allowing organizations to define custom annotation types (FAQs, intents, entities, or domain-specific categories) through configuration rather than code changes. Deployed at JP Morgan Chase, MAFA has eliminated a 1 million utterance backlog while achieving, on average, 86% agreement with human annotators, annually saving over 5,000 hours of manual annotation work. The system processes utterances with annotation confidence classifications, which are typically 85% high, 10% medium, and 5% low across all datasets we tested. This enables human annotators to focus exclusively on ambiguous and low-coverage cases. We demonstrate MAFA's effectiveness across multiple datasets and languages, showing consistent improvements over traditional and single-agent annotation baselines: 13.8% higher Top-1 accuracy, 15.1% improvement in Top-5 accuracy, and 16.9% better F1 in our internal intent classification dataset and similar gains on public benchmarks. This work bridges the gap between theoretical multi-agent systems and practical enterprise deployment, providing a blueprint for organizations facing similar annotation challenges.

MAFA: A Multi-Agent Framework for Enterprise-Scale Annotation with Configurable Task Adaptation

TL;DR

MAFA presents a production-deployed, configurable multi-agent framework for enterprise annotation that combines specialized agents, structured reasoning, and a judge-based consensus to achieve high accuracy at scale. By enabling dynamic task adaptation and integrating with human-in-the-loop workflows, it eliminates large backlogs and delivers measurable business benefits in a banking context. Key innovations include a configuration-driven architecture, ARQ-based structured prompts, few-shot diversity, embedding-enhanced retrieval, and a robust consensus mechanism that preserves auditability and compliance. The approach demonstrates strong cross-domain and multilingual performance, practical latency and scalability, and meaningful operational impact, offering a blueprint for enterprise AI deployments facing annotation bottlenecks.

Abstract

We present MAFA (Multi-Agent Framework for Annotation), a production-deployed system that transforms enterprise-scale annotation workflows through configurable multi-agent collaboration. Addressing the critical challenge of annotation backlogs in financial services, where millions of customer utterances require accurate categorization, MAFA combines specialized agents with structured reasoning and a judge-based consensus mechanism. Our framework uniquely supports dynamic task adaptation, allowing organizations to define custom annotation types (FAQs, intents, entities, or domain-specific categories) through configuration rather than code changes. Deployed at JP Morgan Chase, MAFA has eliminated a 1 million utterance backlog while achieving, on average, 86% agreement with human annotators, annually saving over 5,000 hours of manual annotation work. The system processes utterances with annotation confidence classifications, which are typically 85% high, 10% medium, and 5% low across all datasets we tested. This enables human annotators to focus exclusively on ambiguous and low-coverage cases. We demonstrate MAFA's effectiveness across multiple datasets and languages, showing consistent improvements over traditional and single-agent annotation baselines: 13.8% higher Top-1 accuracy, 15.1% improvement in Top-5 accuracy, and 16.9% better F1 in our internal intent classification dataset and similar gains on public benchmarks. This work bridges the gap between theoretical multi-agent systems and practical enterprise deployment, providing a blueprint for organizations facing similar annotation challenges.
Paper Structure (87 sections, 5 figures, 11 tables)

This paper contains 87 sections, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Problem Context and Business Motivation
  3. The Enterprise Annotation Challenge
  4. Requirements for Production Deployment
  5. Related Work
  6. Annotation Systems and Active Learning
  7. LLM-Based Annotation
  8. Multi-Agent Systems
  9. Structured Reasoning in LLMs
  10. Human-AI Collaboration in Annotation
  11. System Architecture
  12. Configuration-Driven Architecture
  13. Query Planning and Expansion
  14. Specialized Annotation Agents
  15. Structured Agent Prompting
  16. ...and 72 more sections

Figures (5)

  • Figure 1: Architecture of MAFA. The query planning agent analyzes and expands user utterances, which are then processed by multiple specialized ranker agents operating in parallel to generate candidate annotations. A judge agent performs final evaluation to produce optimized results.
  • Figure 2: Accuracy vs latency tradeoff across agent configurations
  • Figure 3: Ablation study visualization showing Top-1 accuracy degradation as components are removed. Error bars indicate standard deviation across 10 runs.
  • Figure 4: Statistical significance heatmap showing p-values for pairwise comparisons. Values below 0.05 (shown in bold) indicate statistically significant differences.
  • Figure 5: Multi-dimensional performance comparison using radar plots across datasets. MAFA (green) consistently outperforms baselines across all metrics.