Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Model

TL;DR

The paper introduces the Latent Instance Difficulty (LID) model to unify training- and inference-time scaling in a solvable last-layer fine-tuning setting with instance-heterogeneous targets. By letting target variance be governed by a heavy-tailed latent precision $\tau_{\mathbf{x}}$, the authors derive a two-tail inference scaling law for pass@$k$ that is training-dependent: the effective exponent $\beta_{\mathrm{eff}}(N)$ grows with training and saturates at the intrinsic tail index $\beta$ set by the difficulty distribution. They provide closed-form predictions for the crossover between finite-$N$ bias-dominated and intrinsic-tail regimes, and a compute-allocation rule that prioritizes training until $\beta_{\mathrm{eff}}(N)$ nears $\beta$, then focuses on inference. The theory is validated through simulations and two real-world proxies: CIFAR-10H (human-label variance) and GSM8K teacher–student distillation, which exhibit the predicted sharpening and saturation of pass@$k$ curves. Overall, the work offers a principled framework for resource allocation across training and inference by linking improvements in generalization to gains in inference-time performance, with broad applicability beyond the linear setting.

Abstract

We analyze neural scaling laws in a solvable model of last-layer fine-tuning where targets have intrinsic, instance-heterogeneous difficulty. In our Latent Instance Difficulty (LID) model, each input's target variance is governed by a latent ``precision'' drawn from a heavy-tailed distribution. While generalization loss recovers standard scaling laws, our main contribution connects this to inference. The pass@$k$ failure rate exhibits a power-law decay, $k^{-β_\text{eff}}$, but the observed exponent $β_\text{eff}$ is training-dependent. It grows with sample size $N$ before saturating at an intrinsic limit $β$ set by the difficulty distribution's tail. This coupling reveals that learning shrinks the ``hard tail'' of the error distribution: improvements in the model's generalization error steepen the pass@$k$ curve until irreducible target variance dominates. The LID model yields testable, closed-form predictions for this behavior, including a compute-allocation rule that favors training before saturation and inference attempts after. We validate these predictions in simulations and in two real-data proxies: CIFAR-10H (human-label variance) and a maths teacher-student distillation task.

Figures (5)

• Figure 1: Training and inference-time scaling laws in the LID setting.Left: Generalization error $\mathcal{L}_{\text{gen}}$ vs. $N$, showing double descent and the two classical regimes. Center: Pass@$k$ inference failure rate $\mathcal{L}_{\text{inf}}(k;N)$ vs. $k$ for several $N>d$, with asymptotic slope $-\beta$ (dashed black) once the mean is well learned. Right: The effective inference exponent $\beta_{\mathrm{eff}}(N)$ extracted from the local log–log slope in a fixed $k$-window. The dotted line marks the asymptote $\beta$; the solid curve is the empirical fit $\beta_{\mathrm{eff}}(N)=\beta - \Delta / (1 + c_\beta N^{\nu})$. We use $\lambda=10^{-9}$ and $\sigma_\eta=10^{-3}$.
• Figure 2: Compute allocation with a training-dependent inference exponent.Left: empirical finite-$N$ case. We plot $\log_{10}\mathcal{L}_{\mathrm{tot}}(N,k;R)$ with $R$ the training:inference weight ratio and $C=Nc_N+kc_k$. Black: grid optimum $\tilde{N}_\star$; white dashed: analytic approximation from equation \ref{['eq:opt_cond_general']}, using $\beta_{\mathrm{eff}}(N)$ (local log--log slope of $\mathcal{L}_{\text{inf}}$) and its discrete derivative. Right: constant-$\beta$ baseline calibrated from the same data; white dashed is the closed-form analytic optimum. Contours clearly shift toward larger $N$ in the finite-$N$ panel, consistent with the $-\beta_{\mathrm{eff}}'(N)\log(\cdot)$ correction.
• Figure 3: Training and inference scaling on CIFAR-10H (frozen backbone, linear head).Top row:Left: Generalization loss $\mathcal{L}_{\text{gen}}(N)$ with the $N^{-1}$ tail in the classical regime. Center: Pass@$k$ failure $\mathcal{L}_{\text{inf}}(k;N)$ for several $N$ on CIFAR-10H; dashed shows a $k^{-\beta}$ reference anchored on the largest-$N$ curve. Right: Effective inference slope $\beta_{\mathrm{eff}}(N)$ estimated from the local log–log slope; the saturating fit $\beta_{\mathrm{eff}}(N)=\beta-\Delta/(1+c_\beta N^\nu)$ is overlaid, approaching the intrinsic tail index $\beta$. Bottom row: A CIFAR‑10 image, its human label distribution, and the Gaussian PDF used to sample training/inference labels, illustrating instance difficulty via label variance.
• Figure 4: GSM8K distillation (teacher$\to$student).Left: strict greedy valid‑only MSE vs. $N$ (1% Winsorized), with a $N^{-1}$ reference. Center:$\mathcal{L}_{\text{inf}}(k;N)$ vs. $k$ for three $N$ values. Right: effective inference slope $\hat{\beta}_{\mathrm{eff}}(N)$, which increases with $N$ and plateaus, consistent with the LID two‑tail law.
• Figure 5: Examples for low and high variance images from CIFAR-10H.Top to bottom: low variance samples are easier to predict (more localized near the average prediction) while high variance samples are difficult.

Theorems & Definitions (10)

• Definition 3.1: Latent Instance Difficulty (LID) Model
• Corollary 3.2
• Proposition 4.2: Bias-free pass@$k$ scaling
• Theorem 4.3: Two-tail mixture law for pass@$k$
• Corollary 4.4: Training‑dependent effective exponent
• Proposition 4.5: Optimal allocation with training-dependent $\beta_\mathrm{eff}(N)$
• Proposition E.3: Two-tail law for $S_N$
• proof : Sketch
• Corollary E.4: Mixture law for $\mathcal{L}_{\text{inf}}(k;N)$
• Corollary E.5: Training-dependent effective exponent, precise form