Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference

Arindam Khaled

TL;DR

This work proposes "Pyramid MoA", a hierarchical Mixture-of-Agents architecture that uses a lightweight Router to dynamically escalate queries only when necessary, and demonstrates that the system introduces negligible latency overhead and allows for a tunable trade-off between performance and budget.

Abstract

Large Language Models (LLMs) face a persistent trade-off between inference cost and reasoning capability. While "Oracle" models (e.g., Llama-3-70B) achieve state-of-the-art accuracy, they are prohibitively expensive for high-volume deployment. Smaller models (e.g., 8B parameters) are cost-effective but struggle with complex tasks. In this work, we propose "Pyramid MoA", a hierarchical Mixture-of-Agents architecture that uses a lightweight Router to dynamically escalate queries only when necessary. By leveraging semantic agreement and confidence calibration among an ensemble of small models, our Router identifies "hard" problems with high precision. On the GSM8K benchmark, our system achieves 93.0% accuracy, effectively matching the Oracle baseline (98.0%) while reducing compute costs by 61%. We demonstrate that the system introduces negligible latency overhead (+0.82s) and allows for a tunable trade-off between performance and budget.

Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference

TL;DR

This work proposes "Pyramid MoA", a hierarchical Mixture-of-Agents architecture that uses a lightweight Router to dynamically escalate queries only when necessary, and demonstrates that the system introduces negligible latency overhead and allows for a tunable trade-off between performance and budget.

Abstract

Large Language Models (LLMs) face a persistent trade-off between inference cost and reasoning capability. While "Oracle" models (e.g., Llama-3-70B) achieve state-of-the-art accuracy, they are prohibitively expensive for high-volume deployment. Smaller models (e.g., 8B parameters) are cost-effective but struggle with complex tasks. In this work, we propose "Pyramid MoA", a hierarchical Mixture-of-Agents architecture that uses a lightweight Router to dynamically escalate queries only when necessary. By leveraging semantic agreement and confidence calibration among an ensemble of small models, our Router identifies "hard" problems with high precision. On the GSM8K benchmark, our system achieves 93.0% accuracy, effectively matching the Oracle baseline (98.0%) while reducing compute costs by 61%. We demonstrate that the system introduces negligible latency overhead (+0.82s) and allows for a tunable trade-off between performance and budget.
Paper Structure (13 sections, 3 equations, 4 figures, 1 table)

This paper contains 13 sections, 3 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. System Components
  4. Decision-Theoretic Routing
  5. Experimental Results
  6. Router Configuration Analysis and Empirical Pivot
  7. Experiment I: Code Generation (Consensus Architecture)
  8. Safety and Mechanism
  9. Efficiency vs. Baselines
  10. Experiment II: Mathematical Reasoning (Anytime Architecture)
  11. The Anytime Pareto Frontier
  12. Related Work
  13. Discussion & Conclusion

Figures (4)

  • Figure 1: Refined Pyramid MoA Schematic: The system extracts ensemble-wide features (semantic agreement and variance) from the base models to estimate $P_{fail}$.
  • Figure 2: Consensus Mechanism: Evaluation on MBPP showing that peer-agreement signals significantly outperform intrinsic model confidence.
  • Figure 3: Code Generation Benchmark (HumanEval): The Consensus MoA matches the upper-bound accuracy of Llama-70B while significantly undercutting the cost of the FrugalGPT baseline.
  • Figure 4: Math Reasoning Frontier (GSM8K): The Anytime Router allows the system to 'short-circuit' calculation for confident answers, strictly increasing quality with depth.