Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling

Yao-Ching Yu, Chun-Chih Kuo, Ziqi Ye, Yu-Cheng Chang, Yueh-Se Li

TL;DR

This paper treats the Generation of each token by LLMs as a Classification (GaC) for ensembling, which fully exploits the probability information at each generation step and better prevents LLMs from producing early incorrect tokens that lead to snowballing errors.

Abstract

Ensembling multiple models has always been an effective approach to push the limits of existing performance and is widely used in classification tasks by simply averaging the classification probability vectors from multiple classifiers to achieve better accuracy. However, in the thriving open-source Large Language Model (LLM) community, ensembling methods are rare and typically limited to ensembling the full-text outputs of LLMs, such as selecting the best output using a ranker, which leads to underutilization of token-level probability information. In this paper, we treat the Generation of each token by LLMs as a Classification (GaC) for ensembling. This approach fully exploits the probability information at each generation step and better prevents LLMs from producing early incorrect tokens that lead to snowballing errors. In experiments, we ensemble state-of-the-art LLMs on several benchmarks, including exams, mathematics and reasoning, and observe that our method breaks the existing community performance ceiling. Furthermore, we observed that most of the tokens in the answer are simple and do not affect the correctness of the final answer. Therefore, we also experimented with ensembling only key tokens, and the results showed better performance with lower latency across benchmarks.

Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling

TL;DR

This paper treats the Generation of each token by LLMs as a Classification (GaC) for ensembling, which fully exploits the probability information at each generation step and better prevents LLMs from producing early incorrect tokens that lead to snowballing errors.

Abstract

Ensembling multiple models has always been an effective approach to push the limits of existing performance and is widely used in classification tasks by simply averaging the classification probability vectors from multiple classifiers to achieve better accuracy. However, in the thriving open-source Large Language Model (LLM) community, ensembling methods are rare and typically limited to ensembling the full-text outputs of LLMs, such as selecting the best output using a ranker, which leads to underutilization of token-level probability information. In this paper, we treat the Generation of each token by LLMs as a Classification (GaC) for ensembling. This approach fully exploits the probability information at each generation step and better prevents LLMs from producing early incorrect tokens that lead to snowballing errors. In experiments, we ensemble state-of-the-art LLMs on several benchmarks, including exams, mathematics and reasoning, and observe that our method breaks the existing community performance ceiling. Furthermore, we observed that most of the tokens in the answer are simple and do not affect the correctness of the final answer. Therefore, we also experimented with ensembling only key tokens, and the results showed better performance with lower latency across benchmarks.
Paper Structure (19 sections, 5 equations, 5 figures, 11 tables)

This paper contains 19 sections, 5 equations, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Analysis and Prior Work
  3. Proposed Method
  4. Overall Process of GaC
  5. Creating the Union Mapping
  6. GaC Ensembling
  7. Ensembling Key Tokens with Threshold
  8. Experiments
  9. Overview
  10. Experimental Settings
  11. Exploring Ensemble Parameters
  12. Comparison with Other Methods
  13. Breaking the Ceiling
  14. Ensemble with Threshold
  15. Conclusion
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Motivation of GaC. The upper part shows CV classification ensemble, while the lower part illustrates ensemble at one text generation step.
  • Figure 2: The rate of identical tokenization for Oxford 5000 common words between different LLMs.
  • Figure 3: Overview of GaC. The left side shows the creation of the mapping matrix, and the right side shows the ensembling during text generation with two LLMs.
  • Figure 4: Results of GaC ensemble with different weights for models from Tab.\ref{['table2']}. Smaller models ensembles on top, larger ones on bottom. The x-axis shows names participating in the ensemble (abbreviated).
  • Figure 5: Ensemble results of CV models with different accuracy gaps on ImageNet. Models' accuracies are next to their names. Each cell shows the ensemble accuracy, with the improvement over the best single model in parentheses.