Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Confidence-Credibility Aware Weighted Ensembles of Small LLMs Outperform Large LLMs in Emotion Detection

Menna Elgabry, Ali Hamdi

TL;DR

The paper tackles emotion detection under data imbalance by proposing a Condorcet-inspired, confidence-credibility weighted ensemble of five small, fully fine-tuned sLLMs (BERT-family). It leverages instance-level confidence and dataset-level credibility to fuse predictions, achieving a macro-F1 of 93.5% on the DAIR-AI dataset while using only 595M parameters, outperforming zero-shot and LoRA-adapted large LLMs. The results demonstrate strong parameter efficiency and robustness, challenging the notion that larger models are always superior for specialized tasks. The approach offers a transferable framework for designing efficient, diverse ensembles in NLP tasks beyond emotion recognition.

Abstract

This paper introduces a confidence-weighted, credibility-aware ensemble framework for text-based emotion detection, inspired by Condorcet's Jury Theorem (CJT). Unlike conventional ensembles that often rely on homogeneous architectures, our approach combines architecturally diverse small transformer-based large language models (sLLMs) - BERT, RoBERTa, DistilBERT, DeBERTa, and ELECTRA, each fully fine-tuned for emotion classification. To preserve error diversity, we minimize parameter convergence while taking advantage of the unique biases of each model. A dual-weighted voting mechanism integrates both global credibility (validation F1 score) and local confidence (instance-level probability) to dynamically weight model contributions. Experiments on the DAIR-AI dataset demonstrate that our credibility-confidence ensemble achieves a macro F1 score of 93.5 percent, surpassing state-of-the-art benchmarks and significantly outperforming large-scale LLMs, including Falcon, Mistral, Qwen, and Phi, even after task-specific Low-Rank Adaptation (LoRA). With only 595M parameters in total, our small LLMs ensemble proves more parameter-efficient and robust than models up to 7B parameters, establishing that carefully designed ensembles of small, fine-tuned models can outperform much larger LLMs in specialized natural language processing (NLP) tasks such as emotion detection.

Confidence-Credibility Aware Weighted Ensembles of Small LLMs Outperform Large LLMs in Emotion Detection

TL;DR

The paper tackles emotion detection under data imbalance by proposing a Condorcet-inspired, confidence-credibility weighted ensemble of five small, fully fine-tuned sLLMs (BERT-family). It leverages instance-level confidence and dataset-level credibility to fuse predictions, achieving a macro-F1 of 93.5% on the DAIR-AI dataset while using only 595M parameters, outperforming zero-shot and LoRA-adapted large LLMs. The results demonstrate strong parameter efficiency and robustness, challenging the notion that larger models are always superior for specialized tasks. The approach offers a transferable framework for designing efficient, diverse ensembles in NLP tasks beyond emotion recognition.

Abstract

This paper introduces a confidence-weighted, credibility-aware ensemble framework for text-based emotion detection, inspired by Condorcet's Jury Theorem (CJT). Unlike conventional ensembles that often rely on homogeneous architectures, our approach combines architecturally diverse small transformer-based large language models (sLLMs) - BERT, RoBERTa, DistilBERT, DeBERTa, and ELECTRA, each fully fine-tuned for emotion classification. To preserve error diversity, we minimize parameter convergence while taking advantage of the unique biases of each model. A dual-weighted voting mechanism integrates both global credibility (validation F1 score) and local confidence (instance-level probability) to dynamically weight model contributions. Experiments on the DAIR-AI dataset demonstrate that our credibility-confidence ensemble achieves a macro F1 score of 93.5 percent, surpassing state-of-the-art benchmarks and significantly outperforming large-scale LLMs, including Falcon, Mistral, Qwen, and Phi, even after task-specific Low-Rank Adaptation (LoRA). With only 595M parameters in total, our small LLMs ensemble proves more parameter-efficient and robust than models up to 7B parameters, establishing that carefully designed ensembles of small, fine-tuned models can outperform much larger LLMs in specialized natural language processing (NLP) tasks such as emotion detection.

Paper Structure

This paper contains 14 sections, 7 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Transformers Architectures for Text Classification
  4. Ensemble Learning and Condorcet's Jury Theorem
  5. Related Work
  6. Methodology
  7. Dataset Description and Preprocessing
  8. Emotion Classification
  9. Results
  10. Performance of the Proposed Ensemble Model
  11. Analysis of Parameter Efficiency
  12. Benchmark Comparison with State-of-the-Art
  13. Limitations and Future Work
  14. Conclusion

Figures (2)

  • Figure 1: System Architecture.
  • Figure 2: Trainer and test architecture.