More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)

TL;DR

This work introduces Reset-and-Discard (ReD), a restart-based query policy that cycles through tasks and discards solved ones to maximize coverage@cost under a fixed budget. By connecting pass@k to coverage@cost via renewal theory, it shows that standard solve-to-completion yields diminishing returns when $1-\text{pass@}k$ follows a power-law with exponent $0<\alpha<1$, while ReD enforces linear growth in many regimes, with the optimal resetting every attempt ($\tau=1$). It also provides methods to predict budget savings from pass@k and to infer the power-law exponent from ReD trajectories when large-$k$ evaluation is costly, validated on multiple LLMs with HumanEval. The approach yields substantial throughput and cost reductions, enabling better deployment economics (Goodput) and offering a framework for inference-power-law estimation in verifiable tasks. Practically, ReD sorts queries by difficulty, allowing smaller models to handle easier problems and reserving larger models for the remaining hard cases, which can significantly reduce token and USD expenditure while preserving accuracy.

Abstract

The performance of large language models (LLMs) on verifiable tasks is usually measured by pass@k, the probability of answering a question correctly at least once in k trials. At a fixed budget, a more suitable metric is coverage@cost, the average number of unique questions answered as a function of the total number of attempts. We connect the two metrics and show that the empirically-observed power-law behavior in pass@k leads to a sublinear growth of the coverage@cost (diminishing returns). To solve this problem, we propose Reset-and-Discard (ReD), a query method of LLMs that increases coverage@cost for any given budget, regardless of the pass@k form. Moreover, given a pass@k, we can quantitatively predict the savings in the total number of attempts using ReD. If pass@k is not available for the model, ReD can infer its power-law exponent. Experiments on three LLMs using HumanEval demonstrate that ReD substantially reduces the required attempts, tokens, and USD cost to reach a desired coverage, while also offering an efficient way to measure inference power-laws.

Figures (4)

• Figure 1: coverage@cost normalized by the total number of questions, $N=164$, against cumulative number of attempts for ReD and the standard (solve-to-completion) evaluation protocol for three models: (Left) llama-3.1-8b-instant, (Middle) llama-3.3-70b-versatile, and (Right) gpt-oss-20b. The dashed lines and shaded regions represent the mean and standard deviation over random realizations, respectively. The dashed-dotted line represents predictions using \ref{['eq:predict_time_coverage_finite_t', 'eq:predict_coverage_finite_t']}. The dotted line shows pass@$1\cdot t$, which is the expected behavior in the first ReD round.
• Figure 2: coverage@cost normalized by the total number of questions, $N=164$, of llama-3.1-8b-instant (standard, solve-to-completion versus ReD) compared to llama-70b-versatile and gpt-oss-20b (standard, solve-to-completion only), measured by (Left) Attempts, (Middle) Tokens, and (Right) USD cost. The dashed lines and shaded regions represent the mean and standard deviation over random realizations, respectively.
• Figure 3: The ratio $-\langle R_n\rangle/\langle R_{n+1} - R_n\rangle$ as a function of the round number $n$ for llama-3.1-8b-instant, where $R_n$ is the number of unsolved problems at round $n$.
• Figure S1: A linear fit of $\log(1-\text{pass@}k)$, of the llama-3.1-8b-instant, versus $\log k$. For $k\gg1$ indeed $(1-\text{pass@}k)\propto k^{-\alpha}$, and the obtained power-law exponent is $\alpha=0.34$.

Theorems & Definitions (5)

• Definition 3.1
• Theorem 4.1: Optimality of ReD
• proof
• Corollary 2.1
• proof