AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs

TL;DR

AdaFuse tackles the rigidity of fixed-granularity inference-time ensembling by introducing an adaptive, word-level ensemble framework guided by decoding uncertainty. It combines a candidate word proposal, an adaptive word commitment rule, and a diversity-aware two-stage search with cross-model scoring to select the best word-span continuation in real time. Empirical results across six benchmarks in QA, arithmetic reasoning, and machine translation show consistent gains over strong baselines, with an average relative improvement of 6.88%. While requiring access to token-level likelihoods, AdaFuse demonstrates a favorable balance between accuracy and efficiency and highlights the value of adaptive control for coordinating heterogeneous models during generation.

Abstract

Large language models (LLMs) exhibit complementary strengths arising from differences in pretraining data, model architectures, and decoding behaviors. Inference-time ensembling provides a practical way to combine these capabilities without retraining. However, existing ensemble approaches suffer from fundamental limitations. Most rely on fixed fusion granularity, which lacks the flexibility required for mid-generation adaptation and fails to adapt to different generation characteristics across tasks. To address these challenges, we propose AdaFuse, an adaptive ensemble decoding framework that dynamically selects semantically appropriate fusion units during generation. Rather than committing to a fixed granularity, AdaFuse adjusts fusion behavior on the fly based on the decoding context, with words serving as basic building blocks for alignment. To be specific, we introduce an uncertainty-based criterion to decide whether to apply ensembling at each decoding step. Under confident decoding states, the model continues generation directly. In less certain states, AdaFuse invokes a diversity-aware scaling strategy to explore alternative candidate continuations and inform ensemble decisions. This design establishes a synergistic interaction between adaptive ensembling and test-time scaling, where ensemble decisions guide targeted exploration, and the resulting diversity in turn strengthens ensemble quality. Experiments on open-domain question answering, arithmetic reasoning, and machine translation demonstrate that AdaFuse consistently outperforms strong ensemble baselines, achieving an average relative improvement of 6.88%. The code is available at https://github.com/CCM0111/AdaFuse.

Figures (7)

• Figure 1: Illustration of the AdaFuse framework. AdaFuse ensembles multiple LLMs during decoding by dynamically adjusting fusion behavior. It expands in confident decoding states and applies diversity-aware scaling under uncertainty. Candidate word sequences are unified and selected using NLL scores, enabling adaptive and stable generation.
• Figure 2: Comparison of Fixed-Length Decoding and AdaFuse on Natural Questions Dataset.
• Figure 3: A Qualitative Example from Natural Questions: Fixed-Length Decoding vs. AdaFuse
• Figure 4: Diversity-Aware Ensemble Scaling Analysis
• Figure 5: End-to-end runtime comparison of ensemble decoding methods on the Natural Questions (NQ) dataset under a unified multi-GPU inference setting.