Leveraging LLM Inconsistency to Boost Pass@k Performance

TL;DR

This work reframes LLM inconsistency across semantically equivalent prompts as a productive signal by introducing the Variator agent, which generates k diverse task variants and submits one solution per variant to boost Pass@k performance. The authors provide theoretical guarantees under a symmetric variant model showing that Pass@k with Variator can outperform a baseline Repeater, especially for larger k, and validate the approach empirically on coding (APPS) and cybersecurity (CTF) tasks using frontier reasoning models. They demonstrate the persistence of the inconsistency effect in OpenAI o3-mini and Claude 3.7 Sonnet across coding and cyber domains, even when variants are equivalent, and show that memorization on public benchmarks can impede gains though private data reveal stronger improvements. The work suggests a practical direction for leveraging LLM brittleness to enhance performance and motivates further research into automatic variant verification and broader domain applicability.

Abstract

Large language models (LLMs) achieve impressive abilities in numerous domains, but exhibit inconsistent performance in response to minor input changes. Rather than view this as a drawback, in this paper we introduce a novel method for leveraging models' inconsistency to boost Pass@k performance. Specifically, we present a "Variator" agent that generates k variants of a given task and submits one candidate solution for each one. Our variant generation approach is applicable to a wide range of domains as it is task agnostic and compatible with free-form inputs. We demonstrate the efficacy of our agent theoretically using a probabilistic model of the inconsistency effect, and show empirically that it outperforms the baseline on the APPS dataset. Furthermore, we establish that inconsistency persists even in frontier reasoning models across coding and cybersecurity domains, suggesting our method is likely to remain relevant for future model generations.

Figures (5)

• Figure 1: Overview of the proposed technique, which is shown both empirically (for a commonly used benchmark) and theoretically (under appropriate symmetry conditions) to improve the $\textrm{Pass@}k$ success rate relative to the standard approach.
• Figure 2: Variant success rate distributions. Blue bars represent the observed histogram of variant success rates. Orange lines represent the expected distribution under the null hypothesis that there is no inconsistency effect in the given setting.
• Figure 3: Distributions of success rates for 25 variants of the same challenge, with three guidance levels. The considerable overlap between the three distributions illustrates that some unguided variants have higher success rates than other guided variants, highlighting the significant performance impact of equivalent variants.
• Figure 4: Success rates of a challenge and its variants.
• Figure 5: Average variant success rate $p_v$ as a function of the original challenge success rate $p_o$ for different models. Orange circles represent individual challenges, and the solid orange line is a smoothed spline interpolation thereof. Model memorization is one likely reason for reduced performance on the public dataset. This effect is no longer in play in the private dataset.

Theorems & Definitions (2)

• Theorem 1
• proof