Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wisdom and Delusion of LLM Ensembles for Code Generation and Repair

Fernando Vallecillos-Ruiz, Max Hort, Leon Moonen

TL;DR

The paper tackles whether ensembles of diverse LLMs can outperform the best single model in software engineering tasks such as code generation and automatic program repair. It empirically analyzes ten instruction-tuned LLMs across three benchmarks, quantifies model complementarity, and defines an ensemble ceiling that can reach up to 83% more problems solved than the top individual model. A key finding is that naive consensus strategies fall into a popularity trap, while a diversity-based selection approach consistently realizes up to 95% of the theoretical potential, enabling effective and cost-efficient ensembles, including small two-model setups. The work provides practical guidance for building robust, scalable LLM ensembles, and releases replication materials to enable broader adoption and further research.

Abstract

Today's pursuit of a single Large Language Model (LMM) for all software engineering tasks is resource-intensive and overlooks the potential benefits of complementarity, where different models contribute unique strengths. However, the degree to which coding LLMs complement each other and the best strategy for maximizing an ensemble's potential are unclear, leaving practitioners without a clear path to move beyond single-model systems. To address this gap, we empirically compare ten individual LLMs from five families, and three ensembles of these LLMs across three software engineering benchmarks covering code generation and program repair. We assess the complementarity between models and the performance gap between the best individual model and the ensembles. Next, we evaluate various selection heuristics to identify correct solutions from an ensemble's candidate pool. We find that the theoretical upperbound for an ensemble's performance can be 83% above the best single model. Our results show that consensus-based strategies for selecting solutions fall into a "popularity trap," amplifying common but incorrect outputs. In contrast, a diversity-based strategy realizes up to 95% of this theoretical potential, and proves effective even in small two-model ensembles, enabling a cost-efficient way to enhance performance by leveraging multiple LLMs.

Wisdom and Delusion of LLM Ensembles for Code Generation and Repair

TL;DR

The paper tackles whether ensembles of diverse LLMs can outperform the best single model in software engineering tasks such as code generation and automatic program repair. It empirically analyzes ten instruction-tuned LLMs across three benchmarks, quantifies model complementarity, and defines an ensemble ceiling that can reach up to 83% more problems solved than the top individual model. A key finding is that naive consensus strategies fall into a popularity trap, while a diversity-based selection approach consistently realizes up to 95% of the theoretical potential, enabling effective and cost-efficient ensembles, including small two-model setups. The work provides practical guidance for building robust, scalable LLM ensembles, and releases replication materials to enable broader adoption and further research.

Abstract

Today's pursuit of a single Large Language Model (LMM) for all software engineering tasks is resource-intensive and overlooks the potential benefits of complementarity, where different models contribute unique strengths. However, the degree to which coding LLMs complement each other and the best strategy for maximizing an ensemble's potential are unclear, leaving practitioners without a clear path to move beyond single-model systems. To address this gap, we empirically compare ten individual LLMs from five families, and three ensembles of these LLMs across three software engineering benchmarks covering code generation and program repair. We assess the complementarity between models and the performance gap between the best individual model and the ensembles. Next, we evaluate various selection heuristics to identify correct solutions from an ensemble's candidate pool. We find that the theoretical upperbound for an ensemble's performance can be 83% above the best single model. Our results show that consensus-based strategies for selecting solutions fall into a "popularity trap," amplifying common but incorrect outputs. In contrast, a diversity-based strategy realizes up to 95% of this theoretical potential, and proves effective even in small two-model ensembles, enabling a cost-efficient way to enhance performance by leveraging multiple LLMs.
Paper Structure (26 sections, 3 equations, 4 figures, 6 tables)

This paper contains 26 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experiment Design
  4. Research Questions
  5. Models and Ensembles
  6. Benchmarks
  7. Implementation
  8. Family Advantage Index
  9. Family Advantage Index
  10. Selection Metrics
  11. Candidate Pool Construction
  12. Selection Strategies
  13. Evaluation Metric
  14. Results and Discussion
  15. Results of RQ1
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Conceptual overview of the selection process from an LLM ensemble's output pool where only 4 programs can be selected per problem.
  • Figure 2: [RQ1.1] Venn diagram of problems solved for Defects4J benchmark for two sets of models indicating their complementarity.
  • Figure 3: [RQ2.2] Comparison of two-model ensembles with Naive strategy vs. the best score of the two models. The heatmaps show the difference in the number of plausible candidates per benchmark (e.g. a negative score indicates that the Naive ensemble strategy performs worse than picking the better single model).
  • Figure 4: [RQ2.2] Comparison of two-model ensembles with diversity-based strategy using CodeBLEU vs. Naive strategy. The heatmaps show the difference in the number of plausible candidates per benchmark (e.g. a negative score indicates that the CodeBLEU diversity-based strategy performs worse than the Naive ensemble strategy).