Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations

William Lugoloobi, Thomas Foster, William Bankes, Chris Russell

TL;DR

The paper investigates whether LLMs encode their own likelihood of success in pre-generation activations and whether this signal can guide more efficient inference. By training simple linear probes on pre-generation residual streams, the authors show that both human difficulty (IRT) and model-specific difficulty (expected success under stochastic decoding) are linearly decodable, yet they capture distinct information, with model difficulty diverging from human judgments as reasoning increases. They demonstrate that binary success signals under fixed decoding policies are more predictable than continuous success rates and that probe-guided routing across a pool of models can achieve strong cost-accuracy tradeoffs, including cost reductions of up to ~70% on challenging benchmarks like MATH and AIME. However, probe reliability degrades with extended reasoning, highlighting a bottleneck for adaptive inference that depends on the quality of difficulty estimates. Overall, linear probes on pre-generation activations enable practical routing and reveal a model-relative notion of difficulty that becomes increasingly distinct from human difficulty as model reasoning capabilities grow.

Abstract

Running LLMs with extended reasoning on every problem is expensive, but determining which inputs actually require additional compute remains challenging. We investigate whether their own likelihood of success is recoverable from their internal representations before generation, and if this signal can guide more efficient inference. We train linear probes on pre-generation activations to predict policy-specific success on math and coding tasks, substantially outperforming surface features such as question length and TF-IDF. Using E2H-AMC, which provides both human and model performance on identical problems, we show that models encode a model-specific notion of difficulty that is distinct from human difficulty, and that this distinction increases with extended reasoning. Leveraging these probes, we demonstrate that routing queries across a pool of models can exceed the best-performing model whilst reducing inference cost by up to 70\% on MATH, showing that internal representations enable practical efficiency gains even when they diverge from human intuitions about difficulty. Our code is available at: https://github.com/KabakaWilliam/llms_know_difficulty

LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations

TL;DR

The paper investigates whether LLMs encode their own likelihood of success in pre-generation activations and whether this signal can guide more efficient inference. By training simple linear probes on pre-generation residual streams, the authors show that both human difficulty (IRT) and model-specific difficulty (expected success under stochastic decoding) are linearly decodable, yet they capture distinct information, with model difficulty diverging from human judgments as reasoning increases. They demonstrate that binary success signals under fixed decoding policies are more predictable than continuous success rates and that probe-guided routing across a pool of models can achieve strong cost-accuracy tradeoffs, including cost reductions of up to ~70% on challenging benchmarks like MATH and AIME. However, probe reliability degrades with extended reasoning, highlighting a bottleneck for adaptive inference that depends on the quality of difficulty estimates. Overall, linear probes on pre-generation activations enable practical routing and reveal a model-relative notion of difficulty that becomes increasingly distinct from human difficulty as model reasoning capabilities grow.

Abstract

Running LLMs with extended reasoning on every problem is expensive, but determining which inputs actually require additional compute remains challenging. We investigate whether their own likelihood of success is recoverable from their internal representations before generation, and if this signal can guide more efficient inference. We train linear probes on pre-generation activations to predict policy-specific success on math and coding tasks, substantially outperforming surface features such as question length and TF-IDF. Using E2H-AMC, which provides both human and model performance on identical problems, we show that models encode a model-specific notion of difficulty that is distinct from human difficulty, and that this distinction increases with extended reasoning. Leveraging these probes, we demonstrate that routing queries across a pool of models can exceed the best-performing model whilst reducing inference cost by up to 70\% on MATH, showing that internal representations enable practical efficiency gains even when they diverge from human intuitions about difficulty. Our code is available at: https://github.com/KabakaWilliam/llms_know_difficulty
Paper Structure (42 sections, 4 equations, 4 figures, 10 tables)

This paper contains 42 sections, 4 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Main Contributions.
  3. Related Work
  4. Predicting Difficulty
  5. Two Notions of Difficulty
  6. Human difficulty (IRT).
  7. Model difficulty: Expected success rate.
  8. Model success: Binary outcome under specified decoding.
  9. Experimental Setup
  10. E2H-AMC: Controlled human--model difficulty comparison.
  11. Additional benchmarks for model difficulty.
  12. Linear probe.
  13. Baselines and metrics.
  14. Results
  15. Human and model difficulty are both linearly decodable but encode different information.
  16. ...and 27 more sections

Figures (4)

  • Figure 1: Human and model difficulty diverge with increased reasoning. On E2H-AMC, as the reasoning level in GPT-OSS-20B is increased, difficulty becomes less human-aligned and more model-specific. Left: (A) Alignment between probe-predicted model difficulty and human IRT difficulty decreases with higher reasoning, indicating that correctness-related signals become less linearly accessible as models solve questions that are typically difficult for humans. Right: (B) Despite this divergence, probe-based predictions consistently outperform human difficulty for predicting Maj@5 failure across reasoning modes, demonstrating that internal activations encode a model-relative notion of difficulty that is distinct from human difficulty.
  • Figure 2: Chain-of-thought length tracks human difficulty but diverges from model success. We plot binned chain-of-thought length (total output tokens, log-scale) against expected values (means) of normalized human IRT difficulty, empirical correctness, empirical success rates, and probe-predicted success (SR@5 and Maj@5) for GPT-OSS-20B at low, medium, and high reasoning modes. Across all settings, output length is positively correlated with human difficulty and negatively correlated with both empirical and predicted success. This effect strengthens with increased reasoning mode, indicating that extended reasoning causes models to allocate more computation to problems humans find difficult, even when those problems are unlikely to be failed. thereby decoupling generation length from model-relative uncertainty.
  • Figure 3: Probe-based routing achieves strong performance-cost tradeoffs on MATH.Left (Cascade): Binary routing between Qwen2.5-Math-7B-Instruct and GPT-OSS-20B-medium. The cascade strategy (red curve) substantially outperforms random routing (gray diamond) across the Pareto frontier, matching GPT-OSS-20B-medium accuracy (orange circle) at 17% lower cost. Right (Utility): Model selection from a pool of five models with varying capabilities and costs. The utility router (red curve) achieves a Pareto improvement over all single-model baselines, exceeding GPT-OSS-20B-high accuracy (red circle) while reducing cost by approximately 70%. Both strategies route difficult queries (low $\hat{p}$) to more capable models. Oracle performance (gold star) represents an upper bound with perfect difficulty prediction. All results use maj@5 with $K=5$ generations.
  • Figure 4: Probe-based routing generalizes across diverse reasoning benchmarks.Left (AIME 2025): Utility-based routing on a challenging competition mathematics benchmark. The router (red curve) traces a Pareto frontier that dominates all individual models, matching GPT-OSS-20B-high's 93.3% accuracy (red circle) at approximately 37% lower cost ($1.15 vs $1.75). The oracle (gold star, 95.6%) represents the theoretical upper bound with perfect prediction, while oracle utility (blue dashed line) shows the cost-optimal oracle policy. Our router matches oracle accuracy but at a higher cost. Right (GSM8K): Utility-based routing on a saturated benchmark with models achieving high accuracies. The router (red curve) substantially outperforms random routing (gray diamond) and efficiently identifies the cost-optimal operating point near Math-7B (red crosses, 94.5%). In this saturated regime, the router correctly avoids expensive high-reasoning models (GPT-OSS-20B-high: 94.4% at $2.4) in favor of the cheaper Math-7B with comparable accuracy—demonstrating cost-aware selection rather than simple accuracy maximization. Both benchmarks show that probe-guided routing adapts to difficulty distributions: preferring stronger models when accuracy varies widely (AIME), and selecting efficient models when performance plateaus (GSM8K). All results use maj@5 with $K{=}5$ generations. The full table of results is in the Appendix Section \ref{['app:routing_strategy_results']}.