EMAFusion: A Self-Optimizing System for Seamless LLM Selection and Integration

TL;DR

EMAFusion addresses the cost and reliability challenges of deploying large language models by unifying taxonomy-based routing, a learned router, and cascading execution with multi-judge CASCADE evaluation. It decomposes tasks, routes easy queries via structured taxonomy cues while handling ambiguous cases with data-driven predictions, and progressively escalates to more capable models using judge-informed deferrals. Empirical results show EMAFusion outperforms the best single model by about 2.6 percentage points and achieves substantial cost savings (roughly 3–4x cheaper than an average model; up to ~20x cheaper than GPT-4) while maintaining or improving accuracy. The framework’s hybrid routing and judge-based fusion reduce bias and enhance practical deployment potential, enabling high accuracy under flexible budget constraints.

Abstract

While recent advances in large language models (LLMs) have significantly enhanced performance across diverse natural language tasks, the high computational and financial costs associated with their deployment remain substantial barriers. Existing routing strategies partially alleviate this challenge by assigning queries to cheaper or specialized models, but they frequently rely on extensive labeled data or fragile task-specific heuristics. Conversely, fusion techniques aggregate multiple LLM outputs to boost accuracy and robustness, yet they often exacerbate cost and may reinforce shared biases. We introduce EMAFusion, a new framework that self-optimizes for seamless LLM selection and reliable execution for a given query. Specifically, EMAFusion integrates a taxonomy-based router for familiar query types, a learned router for ambiguous inputs, and a cascading approach that progressively escalates from cheaper to more expensive models based on multi-judge confidence evaluations. Through extensive evaluations, we find EMAFusion outperforms the best individual models by over 2.6 percentage points (94.3% vs. 91.7%), while being 4X cheaper than the average cost. EMAFusion further achieves a remarkable 17.1 percentage point improvement over models like GPT-4 at less than 1/20th the cost. Our combined routing approach delivers 94.3% accuracy compared to taxonomy-based (88.1%) and learned model predictor-based (91.7%) methods alone, demonstrating the effectiveness of our unified strategy. Finally, EMAFusion supports flexible cost-accuracy trade-offs, allowing users to balance their budgetary constraints and performance needs.

Figures (8)

• Figure 1: Overview of the EMAFusion TM pipeline. An incoming query first goes to a Taxonomy Router to check if it belongs to a known class. If so, it is routed directly to a set of suitable models; otherwise, a Learned Router selects a candidate set. Finally, a Cascading Router picks from the candidate models in order of cost/performance, and a series of judges verifies the output.
• Figure 2: Taxonomy-based routing process.
• Figure 3: Comparison of hybrid approach (Taxonomy + Learned) with individual approaches of Taxonomy only and Learned only routing.
• Figure 4: Comparison of cost and accuracy across our proposed EMAFusion TM after cascading with other state-of-the-art models. The average performance is denoted by the horizontal green line.
• Figure 5: Router Model vs Sample level accuracy comparison.