Harnessing Consistency for Robust Test-Time LLM Ensemble

TL;DR

This work addresses robustness gaps in test-time LLM ensembles by identifying token-space misalignments and cross-model disagreements as key failure modes. It introduces CoRE, a plug-and-play framework that combines token-consistency (a low-pass filter on token disparities) and model-consistency (confidence-weighted, cross-model agreement) to produce robust ensemble predictions with no additional inference cost. Extensive experiments across six benchmarks and multiple ensemble strategies show that CoRE yields consistent accuracy gains, reduces negative ensembling, and scales stably as more models are added. The approach significantly improves reliability in real-world ensemble deployments and offers a principled direction for robustness-enhanced LLM ensembles.

Abstract

Different large language models (LLMs) exhibit diverse strengths and weaknesses, and LLM ensemble serves as a promising approach to integrate their complementary capabilities. Despite substantial progress in improving ensemble quality, limited attention has been paid to the robustness of ensembles against potential erroneous signals, which often arise from heterogeneous tokenization schemes and varying model expertise. Our analysis shows that ensemble failures typically arise from both the token level and the model level: the former reflects severe disagreement in token predictions, while the latter involves low confidence and pronounced disparities among models. In light of this, we propose CoRE, a plug-and-play technique that harnesses model consistency for robust LLM ensemble, which can be seamlessly integrated with diverse ensemble methods. Token-level consistency captures fine-grained disagreements by applying a low-pass filter to downweight uncertain tokens with high inconsistency, often due to token misalignment, thereby improving robustness at a granular level. Model-level consistency models global agreement by promoting model outputs with high self-confidence and minimal divergence from others, enhancing robustness at a coarser level. Extensive experiments across diverse benchmarks, model combinations, and ensemble strategies demonstrate that CoRE consistently improves ensemble performance and robustness.

Figures (12)

• Figure 1: Motivation of CoRE. Left: vanilla ensemble yields an incorrect prediction because the token _r, misaligned from the assistant token rine, dominates the ensemble. Right:CoRE penalizes inconsistent tokens, rendering correct prediction r.
• Figure 2: Ensemble performance across six benchmarks. We report the average performance of different ensemble methods with (CoRE) and without (Vanilla) CoRE.
• Figure 3: Overview of CoRE. Token probabilities from different models are first aligned through token mapping, and a reference probability distribution is constructed for consistency evaluation. Token consistency mitigates the influence of inconsistent tokens, while model consistency highlights reliable and self-consistent models. Combining both yields a robust and improved ensemble prediction.
• Figure 4: Observations. (a) Token probability disparity: Aligned tokens (blue) exhibit smaller disparities than misaligned ones (orange), indicating higher consistency. (b) Entropy of token probability: Correct answers (blue) exhibit lower entropy than wrong ones (orange), indicating higher confidence. (c) Sum of token consistency: Correct answers (blue) exhibit higher sum of token consistency score than wrong ones (orange).
• Figure 5: Robustness against noises. CoRE maintains stable performance with minimal degradation under noises.