Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies

Siddhant Kulkarni, Yukta Kulkarni

Abstract

The adoption of large language models (LLMs) for structured information extraction from financial documents has accelerated rapidly, yet production deployments face fundamental architectural decisions with limited empirical guidance. We present a systematic benchmark comparing four multi-agent orchestration architectures: sequential pipeline, parallel fan-out with merge, hierarchical supervisor-worker and reflexive self-correcting loop. These are evaluated across five frontier and open-weight LLMs on a corpus of 10,000 SEC filings (10-K, 10-Q and 8-K forms). Our evaluation spans 25 extraction field types covering governance structures, executive compensation and financial metrics, measured along five axes: field-level F1, document-level accuracy, end-to-end latency, cost per document and token efficiency. We find that reflexive architectures achieve the highest field-level F1 (0.943) but at 2.3x the cost of sequential baselines, while hierarchical architectures occupy the most favorable position on the cost-accuracy Pareto frontier (F1 0.921 at 1.4x cost). We further present ablation studies on semantic caching, model routing and adaptive retry strategies, demonstrating that hybrid configurations can recover 89\% of the reflexive architecture's accuracy gains at only 1.15x baseline cost. Our scaling analysis from 1K to 100K documents per day reveals non-obvious throughput-accuracy degradation curves that inform capacity planning. These findings provide actionable guidance for practitioners deploying multi-agent LLM systems in regulated financial environments.

Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies

Abstract

The adoption of large language models (LLMs) for structured information extraction from financial documents has accelerated rapidly, yet production deployments face fundamental architectural decisions with limited empirical guidance. We present a systematic benchmark comparing four multi-agent orchestration architectures: sequential pipeline, parallel fan-out with merge, hierarchical supervisor-worker and reflexive self-correcting loop. These are evaluated across five frontier and open-weight LLMs on a corpus of 10,000 SEC filings (10-K, 10-Q and 8-K forms). Our evaluation spans 25 extraction field types covering governance structures, executive compensation and financial metrics, measured along five axes: field-level F1, document-level accuracy, end-to-end latency, cost per document and token efficiency. We find that reflexive architectures achieve the highest field-level F1 (0.943) but at 2.3x the cost of sequential baselines, while hierarchical architectures occupy the most favorable position on the cost-accuracy Pareto frontier (F1 0.921 at 1.4x cost). We further present ablation studies on semantic caching, model routing and adaptive retry strategies, demonstrating that hybrid configurations can recover 89\% of the reflexive architecture's accuracy gains at only 1.15x baseline cost. Our scaling analysis from 1K to 100K documents per day reveals non-obvious throughput-accuracy degradation curves that inform capacity planning. These findings provide actionable guidance for practitioners deploying multi-agent LLM systems in regulated financial environments.
Paper Structure (37 sections, 12 figures, 12 tables)

This paper contains 37 sections, 12 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Research Questions
  3. Contributions
  4. Related Work
  5. Multi-Agent LLM Systems
  6. LLM Evaluation and Benchmarking
  7. Financial Document Processing with LLMs
  8. Orchestration Patterns in Production Systems
  9. System Architecture
  10. Architecture A: Sequential Pipeline
  11. Architecture B: Parallel Fan-Out with Merge
  12. Architecture C: Hierarchical Supervisor-Worker
  13. Architecture D: Reflexive Self-Correcting Loop
  14. Experimental Setup
  15. Dataset
  16. ...and 22 more sections

Figures (12)

  • Figure 1: Architecture A: Sequential Pipeline. Documents flow through a fixed chain of agents with cumulative context passing.
  • Figure 2: Architecture B: Parallel Fan-Out with Merge. Independent extraction branches execute concurrently before reconciliation.
  • Figure 3: Architecture C: Hierarchical Supervisor-Worker. The supervisor dynamically allocates tasks and selectively re-extracts low-confidence fields.
  • Figure 4: Architecture D: Reflexive Self-Correcting Loop. Verification failures trigger iterative critique and correction cycles (up to 3 iterations).
  • Figure 5: Per-domain F1 across all five models under the hierarchical architecture. Financial metrics consistently achieve the highest scores due to standardized GAAP formats; executive compensation is the most challenging domain across all models.
  • ...and 7 more figures