Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMBoost: Make Large Language Models Stronger with Boosting

Zehao Chen, Tianxiang Ai, Yifei Li, Gongxun Li, Yuyang Wei, Wang Zhou, Guanghui Li, Bin Yu, Zhijun Chen, Hailong Sun, Fuzhen Zhuang, Jianxin Li, Deqing Wang, Yikun Ban

TL;DR

LLMBoost advances ensemble learning for large language models by moving beyond black-box output fusion to exploit internal representations. It introduces a cross-model attention mechanism that allows successors to fuse predecessors’ hidden states, a chain training paradigm with an error-suppression objective for progressive correction, and a near-parallel inference scheme that pipelines states across models to reduce latency. The authors establish theoretical guarantees for monotonic improvement under bounded correction and demonstrate empirical gains on commonsense and arithmetic reasoning benchmarks, with notable latency reductions and competitive memory use. The results suggest a practical path to more accurate and efficient multi-LLM systems, including industrial tool-scheduling applications like CCAD.

Abstract

Ensemble learning of LLMs has emerged as a promising alternative to enhance performance, but existing approaches typically treat models as black boxes, combining the inputs or final outputs while overlooking the rich internal representations and interactions across models.In this work, we introduce LLMBoost, a novel ensemble fine-tuning framework that breaks this barrier by explicitly leveraging intermediate states of LLMs. Inspired by the boosting paradigm, LLMBoost incorporates three key innovations. First, a cross-model attention mechanism enables successor models to access and fuse hidden states from predecessors, facilitating hierarchical error correction and knowledge transfer. Second, a chain training paradigm progressively fine-tunes connected models with an error-suppression objective, ensuring that each model rectifies the mispredictions of its predecessor with minimal additional computation. Third, a near-parallel inference paradigm design pipelines hidden states across models layer by layer, achieving inference efficiency approaching single-model decoding. We further establish the theoretical foundations of LLMBoost, proving that sequential integration guarantees monotonic improvements under bounded correction assumptions. Extensive experiments on commonsense reasoning and arithmetic reasoning tasks demonstrate that LLMBoost consistently boosts accuracy while reducing inference latency.

LLMBoost: Make Large Language Models Stronger with Boosting

TL;DR

LLMBoost advances ensemble learning for large language models by moving beyond black-box output fusion to exploit internal representations. It introduces a cross-model attention mechanism that allows successors to fuse predecessors’ hidden states, a chain training paradigm with an error-suppression objective for progressive correction, and a near-parallel inference scheme that pipelines states across models to reduce latency. The authors establish theoretical guarantees for monotonic improvement under bounded correction and demonstrate empirical gains on commonsense and arithmetic reasoning benchmarks, with notable latency reductions and competitive memory use. The results suggest a practical path to more accurate and efficient multi-LLM systems, including industrial tool-scheduling applications like CCAD.

Abstract

Ensemble learning of LLMs has emerged as a promising alternative to enhance performance, but existing approaches typically treat models as black boxes, combining the inputs or final outputs while overlooking the rich internal representations and interactions across models.In this work, we introduce LLMBoost, a novel ensemble fine-tuning framework that breaks this barrier by explicitly leveraging intermediate states of LLMs. Inspired by the boosting paradigm, LLMBoost incorporates three key innovations. First, a cross-model attention mechanism enables successor models to access and fuse hidden states from predecessors, facilitating hierarchical error correction and knowledge transfer. Second, a chain training paradigm progressively fine-tunes connected models with an error-suppression objective, ensuring that each model rectifies the mispredictions of its predecessor with minimal additional computation. Third, a near-parallel inference paradigm design pipelines hidden states across models layer by layer, achieving inference efficiency approaching single-model decoding. We further establish the theoretical foundations of LLMBoost, proving that sequential integration guarantees monotonic improvements under bounded correction assumptions. Extensive experiments on commonsense reasoning and arithmetic reasoning tasks demonstrate that LLMBoost consistently boosts accuracy while reducing inference latency.
Paper Structure (51 sections, 13 theorems, 49 equations, 3 figures, 16 tables, 2 algorithms)

This paper contains 51 sections, 13 theorems, 49 equations, 3 figures, 16 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Framework: LLMBoost
  4. Framework Architecture
  5. Cross-Model Attention.
  6. Chain Training Paradigm
  7. Near-Parallel Inference Paradigm
  8. Theoretical Guarantees: Why Sequential Correction Works
  9. Inference Efficacy: Error Reduction Guarantee
  10. Optimization Stability: Guaranteed Descent
  11. Experiments
  12. Main Experiments Results
  13. Conclusion
  14. Symbols used in the text
  15. Broader Impact and Future Applications & Limitations
  16. ...and 36 more sections

Key Result

Theorem 1

Under Assumption ass:correction_efficacy, For each model $i \geq 1$, decoding step $t$, and vocabulary dimension $v$, there exists an upper bound $\lambda_{i, t, v}^* > 0$ for the scaling coefficient. For any weight $\lambda_{i} \in (0, \lambda_{i, t, v}^*)$, the expected squared error of the new en

Figures (3)

  • Figure 1: LLMBoost Framework. The overall framework comprises three key components: (1) LLMBoost Model Design, (2) Training Paradigm, and (3) Error-Suppression Objective.
  • Figure 2: Near-parallel inference paradigm. The figure uses the timeline as the axis to show the computation state of each model at each moment. Hidden states are propagated across layers and models, and logits are aggregated and decoded into the final prediction, enabling overlapped computation and reduced inference latency.
  • Figure 3: Overall system efficiency comparison of LLMBoost and baseline ensemble methods.

Theorems & Definitions (23)

  • Theorem 1
  • Theorem 2: Guaranteed Descent on Primary Loss
  • Lemma 1: Asymptotic Form of Single-Dimension MSE Change
  • proof
  • Theorem 3: Restate
  • proof
  • Corollary 1: Restate
  • proof
  • Definition 1: Hessian as a Symmetric Bilinear Form
  • Definition 2: Operator Norm of the Hessian Tensor
  • ...and 13 more