Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stable LLM Ensemble: Interaction between Example Representativeness and Diversity

Junichiro Niimi

TL;DR

The paper addresses how example representativeness and output diversity affect one-shot LLM ensembles for sentiment analysis. It introduces Centroid-based Representative Examples (CREs) and analyzes the impact of sampling temperature, demonstrating that CREs with a high temperature ($T=1.5$) yield the best ensemble performance (macro-$F_1$ = $0.636$, RMSE = $0.512$), outperforming random exemplars and even 5-shot prompting by notable margins. Using five base models with median aggregation on the Yelp dataset, the study shows CREs + high $T$ strike a favorable balance between representational coverage and diversity, and that self-consistency among base models mediates ensemble gains. The findings offer a practical, domain-independent approach to robust LLM ensembles, with broad applicability to ordinal prediction tasks and potential use of self-consistency as a confidence signal for downstream systems.

Abstract

Large language models (LLMs) have achieved remarkable results in wide range of domains. However, the accuracy and robustness of one-shot LLM predictions remain highly sensitive to the examples and the diversity among ensemble members. This study systematically investigates the effects of example representativeness (one-shot strategy) and output diversity (sampling temperature) on LLM ensemble performance. Two one-shot strategies are compared: centroid-based representative examples (proposed) and randomly sampled examples (baseline) and sampling temperature also is varied. The proposed approach with higher temperature setting significantly outperforms random selection by +7.6% (macro-F1) and -10.5% (RMSE). Furthermore, the proposed model exceeds 5-shot prompting by +21.1% (macro-F1) and -24.0% (RMSE). Our findings demonstrate that combining representative example selection with increased temperature provides the appropriate level of diversity to the ensemble. This work highlights the practical importance of both example selection and controlled diversity in designing effective one-shot LLM ensembles.

Stable LLM Ensemble: Interaction between Example Representativeness and Diversity

TL;DR

The paper addresses how example representativeness and output diversity affect one-shot LLM ensembles for sentiment analysis. It introduces Centroid-based Representative Examples (CREs) and analyzes the impact of sampling temperature, demonstrating that CREs with a high temperature () yield the best ensemble performance (macro- = , RMSE = ), outperforming random exemplars and even 5-shot prompting by notable margins. Using five base models with median aggregation on the Yelp dataset, the study shows CREs + high strike a favorable balance between representational coverage and diversity, and that self-consistency among base models mediates ensemble gains. The findings offer a practical, domain-independent approach to robust LLM ensembles, with broad applicability to ordinal prediction tasks and potential use of self-consistency as a confidence signal for downstream systems.

Abstract

Large language models (LLMs) have achieved remarkable results in wide range of domains. However, the accuracy and robustness of one-shot LLM predictions remain highly sensitive to the examples and the diversity among ensemble members. This study systematically investigates the effects of example representativeness (one-shot strategy) and output diversity (sampling temperature) on LLM ensemble performance. Two one-shot strategies are compared: centroid-based representative examples (proposed) and randomly sampled examples (baseline) and sampling temperature also is varied. The proposed approach with higher temperature setting significantly outperforms random selection by +7.6% (macro-F1) and -10.5% (RMSE). Furthermore, the proposed model exceeds 5-shot prompting by +21.1% (macro-F1) and -24.0% (RMSE). Our findings demonstrate that combining representative example selection with increased temperature provides the appropriate level of diversity to the ensemble. This work highlights the practical importance of both example selection and controlled diversity in designing effective one-shot LLM ensembles.
Paper Structure (34 sections, 2 figures, 6 tables)

This paper contains 34 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Research Gaps
  4. RQ1 Effect of temperature on ensemble effectiveness:
  5. RQ2 Effect of example representativeness on ensemble performance:
  6. RQ3 Optimal combination of representativeness and diversity:
  7. RQ4 Comparison with 5-shot model:
  8. RQ5 Effect of consistency:
  9. Related Works
  10. Sentiment Analysis
  11. Ensemble in LLMs
  12. Consistency and Reliability in LLMs
  13. Proposed Model
  14. Base Models
  15. One-shot Strategies
  16. ...and 19 more sections

Figures (2)

  • Figure 1: Basic Prompt
  • Figure 2: Rating distributions for examples in each group. While the RSEs are relatively balanced across the five strata, the CREs are notably skewed toward higher ratings.