Do Large Language Models Know What They Are Capable Of?

TL;DR

This paper investigates whether large language models can predict their own success on tasks (in-advance confidence), how these predictions evolve during multi-step tasks, and whether learning from in-context experiences improves decision making in high-cost scenarios. Through three experiments, the authors measure calibration (via AUROC), contract-based risk decisions, and confidence updates across intermediate steps in complex tasks. They find systematic overconfidence across models, with some frontier models (notably Claude Sonnet variants) showing calibrated improvements from experience, while others show little or no improvement; higher general capability does not reliably yield better discrimination. Decisions are approximately rational given self-estimated success probabilities, but optimistic estimates lead to suboptimal choices, highlighting a gap in self-awareness of capabilities. The results have implications for AI misuse and misalignment risk and underscore the need for ongoing calibration and safety evaluations as LLMs become more capable.

Abstract

We investigate whether large language models (LLMs) can predict whether they will succeed on a given task and whether their predictions improve as they progress through multi-step tasks. We also investigate whether LLMs can learn from in-context experiences to make better decisions about whether to pursue a task in scenarios where failure is costly. All LLMs we tested are overconfident, but most predict their success with better-than-random discriminatory power. We find that newer and larger LLMs generally do not have greater discriminatory power, though Claude models do show such a trend. On multi-step agentic tasks, the overconfidence of several frontier LLMs worsens as they progress through the tasks, and reasoning LLMs perform comparably to or worse than non-reasoning LLMs. With in-context experiences of failure, some but not all LLMs reduce their overconfidence leading to significantly improved decision making, while others do not. Interestingly, all LLMs' decisions are approximately rational given their estimated probabilities of success, yet their overly-optimistic estimates result in poor decision making. These results suggest that current LLM agents are hindered by their lack of awareness of their own capabilities. We discuss the implications of LLMs' awareness of their capabilities for AI misuse and misalignment risks.

Figures (8)

• Figure 1: Overview of experiments and key results. Top left: Experiment 1, eliciting in-advance confidence estimates on single-step coding tasks. Middle: Experiment 2. Work contracts are offered to the LLM sequentially, and the LLM is prompted for a confidence estimate and accept/decline decision for each contract. Previous contracts, submissions, and outcomes remain in-context, and the LLM can reflect on these experiences when deciding whether to accept new contracts. Bottom left: Experiment 3, eliciting confidence estimates at each intermediate step on multi-step tasks. The prompts and responses shown in the figure are paraphrased. Right: A key result from each experiment. In the top-right figure, the capability score is the average of scores on MBPP mbpp, GPQA rein2024gpqa, MMLU-Pro (100 samples each from math, law, engineering, and health) mmlu_pro, and BigCodeBench zhuo2025bigcodebench.
• Figure 2: Overconfidence and discriminatory power of LLMs on BigCodeBench tasks. (A) Predicted success rate $\frac{1}{N}\sum_{i=1}^N \hat{p}_i$ (circles) and true success rate (stars). (B) Overconfidence (predicted success rate minus true success rate). Note that the overconfidence of Claude models is monotonically decreasing. (C) Area under receiver-operator characteristic curve (AUROC), a measure of LLMs' discriminatory power in distinguishing tasks they can accomplish from those they cannot. Error bars show 95% confidence intervals (method of DeLong1988). Note that the AUROC of Claude models appears to be on an improving trend. For reasoning LLMs (Sonnet 3.7-4.5, Opus 4, and GPT 5.1), the reasoning token budget was set to 0 to force the LLMs to provide in-advance confidence estimates. Sonnet 3.5 and Haiku 3.5 are the 20241022 versions.
• Figure 3: Learning from in-context experiences of success and failure. (A) Performance on the $n$th contract ($n=1,...,9$) of GPT 4.1 (top row) and Claude Sonnet 3.5 (bottom row). Left column: AUROC at contract $n$ calculated from the confidence estimates $\{\hat{p}_{i,n}\}_{i=1}^M$, with 95% CI (shaded). GPT 4.1 improves slightly, but Sonnet 3.5 does not. Middle column: Contract acceptance rate (fraction of contracts accepted across the 512 samples on the $n$th contract) and predicted success rate ($\frac{1}{M}\sum_{i=1}^{M} \hat{p}_{i,n}$). Sonnet 3.5 reaches the perfect baseline contract acceptance rate by contract 5, but GPT 4.1 shows almost no change. Right column: Expected profit on the $n$th contract, estimated as the average profit across samples, with 95% CI (shaded). Sonnet 3.5's success is due to its well-calibrated contract acceptance rate. Appendix \ref{['exp2_data']} shows these data for all other LLMs tested. (B) AUROC on contracts 1 and 9 with 95% CI (shaded). For many LLMs AUROC improves only slightly, and for some it degrades. (C) Contract acceptance rate (circles) and predicted success rate (squares) on contracts 1 and 9. Contract acceptance rates drop more than predicted success rates, indicating positive risk aversion. (D) Expected profit on contracts 1 and 9 with 95% CI (shaded). For reasoning LLMs, the reasoning token budget was set to 0 to force the LLMs to provide in-advance confidence estimates and contract decisions. Sonnet 3.5 and Haiku 3.5 are the 20241022 versions.
• Figure 4: Predicting success at intermediate steps on multi-step SWE-Bench tasks. (A) Predicted success rate after step $s$, $\frac{1}{N}\sum_i\hat{p}_{i,s}$ (solid), and true success rate (stars). All tested LLMs are overconfident, and only GPT 4o significantly reduces its overconfidence. The 'lower is better' label indicates that lower predicted success rates are closer to the true success rates. Reasoning settings are denoted in parentheses: (0) and (4k) indicate 0 and 4096 reasoning token budgets, (none) and (med.) indicate reasoning effort settings of 'none' and 'medium'. (B) Comparison of initial AUROC at step 1 (circles) and after-the-fact AUROC (squares), with 95% CI DeLong1988. Reasoning models perform comparably to or worse than non-reasoning models. (C) Change in AUROC from step 1 to step $n$, and final after-the-fact AUROC (square data point), with 95% CI (shaded). All OpenAI models except GPT 5.1 (med.) improve step-by-step, while Claude models first improve, but then become worse than their initial AUROC. For panel C, confidence intervals are computed with the method for correlated time-series data from DeLong1988.
• Figure 5: Rationality of LLM decision making under self-reported likelihood estimates. Top row: Fraction of contract decisions that adhere to the decision threshold, equal to the average classification accuracy as defined in footnote \ref{['foot']}. The classification accuracy is computed for each value of $w$, and error bars show 2 standard deviations of these values. Middle row: Von Neumann-Morgenstern (vNM) utility functions $u(w)$. Bottom row: Absolute (Arrow-Pratt) risk aversion. The drop in risk aversion for negative $w$ is suggestive of preferences similar to prospect theory kahneman1979prospect.