LLMs Encode How Difficult Problems Are

TL;DR

This work probes whether problem difficulty is internally encoded in LLM representations in a way that aligns with human judgment and how this encoding evolves under RLVR. Using linear probes across 60 models and two Easy2HardBench math subsets (AMC and GSM8K), the authors show human-labeled difficulty is highly decodable (ρ ≈ 0.88) and scales with model size, while LLM-derived difficulty is weaker and less scalable. Steering experiments reveal that moving models toward easier representations reduces hallucination and shortens outputs, often increasing code-generation behavior. During GRPO training, human-aligned difficulty strengthens alongside performance, whereas automated difficulty signals degrade, suggesting RL amplifies a stable human signal but misaligns automated estimates as models improve. Together, these results highlight the value of human difficulty annotations for guiding and evaluating mathematical reasoning in LLMs, and they illustrate how RL-based training can reshape internal difficulty representations toward more effective strategies.

Abstract

Large language models exhibit a puzzling inconsistency: they solve complex problems yet frequently fail on seemingly simpler ones. We investigate whether LLMs internally encode problem difficulty in a way that aligns with human judgment, and whether this representation tracks generalization during reinforcement learning post-training. We train linear probes across layers and token positions on 60 models, evaluating on mathematical and coding subsets of Easy2HardBench. We find that human-labeled difficulty is strongly linearly decodable (AMC: $ρ\approx 0.88$) and exhibits clear model-size scaling, whereas LLM-derived difficulty is substantially weaker and scales poorly. Steering along the difficulty direction reveals that pushing models toward "easier" representations reduces hallucination and improves accuracy. During GRPO training on Qwen2.5-Math-1.5B, the human-difficulty probe strengthens and positively correlates with test accuracy across training steps, while the LLM-difficulty probe degrades and negatively correlates with performance. These results suggest that human annotations provide a stable difficulty signal that RL amplifies, while automated difficulty estimates derived from model performance become misaligned precisely as models improve. We release probe code and evaluation scripts to facilitate replication.

Figures (10)

• Figure 1: Steering towards “easy” reduces hallucination-like repetition in Qwen2.5-Math-1.5B. With $\alpha=-3$, the model switches from repetitive output (baseline / $+\!3$) to concise, code-assisted reasoning and a parsed correct answer ($64$).
• Figure 2: Probe performance scales with model size following a power law. The scatterplot shows the best performing probes for various models on each of the three chosen datasets. Each point represents the best-performing linear probe for a given model. We fit a power law in log space on the gap to perfect probe performance (1 - performance) as a function of model size. From this, we observe a scaling trend that persists across all datasets, but is strongest in E2H subsets with human labels (AMC and Codeforces), rather than GSM8K with LLM labels.
• Figure 3: Optimal probe positions vary by domain. Percentage of best-performing probes at each token position (negative indices from end of prompt). Codeforces shows strong preference for the final token (50% of probes), while mathematical datasets (AMC, GSM8K) distribute more evenly across recent tokens, suggesting that difficulty signals are encoded differently across domains.
• Figure 4: Steering towards "easy" improves mathematical reasoning performance. Pass@1 accuracy on MATH500 for Qwen2.5-Math-1.5B across steering coefficients $\alpha \in [-3, 3]$. Negative coefficients (easier direction) improve performance across all difficulty bins, while positive coefficients (harder direction) degrade performance, particularly on medium and hard problems.
• Figure 5: Specialised fine-tuning shows diminishing returns for larger models. Probe accuracy on Codeforces for Qwen-2.5-Instruct and its specialised variants. Values show accuracy change after task-specific fine-tuning: 7B models improve modestly, while most 32B models degrade slightly.