Beyond the Loss Curve: Scaling Laws, Active Learning, and the Limits of Learning from Exact Posteriors

TL;DR

The paper introduces an oracle-based framework that uses class-conditional normalizing flows to define a synthetic data-generating world with tractable, exact posteriors p(y|x). This enables a precise Bayes-risk decomposition L(q)=E_x[H(p(y|x))]+E_x[KL(p(y|x)||q(y|x)) and reveals that epistemic uncertainty decays as a power law with dataset size even when the total loss plateaus, exposing ongoing learning hidden from standard benchmarks. It demonstrates architectural differences (e.g., ResNet vs ViT), shows that soft labels from exact posteriors improve calibration, and proves that shift type (covariate versus prior) governs robustness more than shift magnitude. The results have practical implications for architecture choice, data-efficient training, robustness evaluation, and active learning, while acknowledging the synthetic nature of the oracle world and the need to carefully translate insights to real data. Overall, the framework provides ground-truth insight into learning dynamics that standard metrics cannot capture, guiding more rigorous evaluation and design of learning systems.

Abstract

How close are neural networks to the best they could possibly do? Standard benchmarks cannot answer this because they lack access to the true posterior p(y|x). We use class-conditional normalizing flows as oracles that make exact posteriors tractable on realistic images (AFHQ, ImageNet). This enables five lines of investigation. Scaling laws: Prediction error decomposes into irreducible aleatoric uncertainty and reducible epistemic error; the epistemic component follows a power law in dataset size, continuing to shrink even when total loss plateaus. Limits of learning: The aleatoric floor is exactly measurable, and architectures differ markedly in how they approach it: ResNets exhibit clean power-law scaling while Vision Transformers stall in low-data regimes. Soft labels: Oracle posteriors contain learnable structure beyond class labels: training with exact posteriors outperforms hard labels and yields near-perfect calibration. Distribution shift: The oracle computes exact KL divergence of controlled perturbations, revealing that shift type matters more than shift magnitude: class imbalance barely affects accuracy at divergence values where input noise causes catastrophic degradation. Active learning: Exact epistemic uncertainty distinguishes genuinely informative samples from inherently ambiguous ones, improving sample efficiency. Our framework reveals that standard metrics hide ongoing learning, mask architectural differences, and cannot diagnose the nature of distribution shift.

Figures (9)

• Figure 1: Normalizing flows enable exact posterior computation on realistic images.(Left) We train class-conditional flows on AFHQ/ImageNet, creating a frozen oracle with tractable likelihoods. (Center) The oracle decomposes prediction error into aleatoric (irreducible) and epistemic (reducible) components, quantities hidden in standard benchmarks. (Right) We apply this framework to diagnose scaling laws, quantify distribution shift with exact KL divergence, and train models with exact soft labels.
• Figure 2: Epistemic error follows a power law even when total loss plateaus.(a) Total cross-entropy decreases with dataset size; MobileNet shows the highest loss, ResNet/ConvNeXt the lowest. (b) Epistemic uncertainty (KL from oracle) follows $N^{-\alpha}$; dashed lines are power-law fits. This decay continues even when total loss appears flat. (c) Accuracy improves from 92--96% at $N{=}100$ toward 97--98% at $N{=}10{,}000$. (d) Scaling exponents reveal architectural differences: MobileNet improves fastest ($\alpha{=}0.135$), ViT stalls without pretraining ($\alpha{=}0.032$).
• Figure 3: What shifts matters more than how much: KL magnitude alone poorly predicts performance ($R^2{=}0.04$).(a) Test accuracy vs. exact KL divergence: class imbalance (blue) maintains ${\sim}$97% accuracy; Gaussian noise (low points) collapses to ${\sim}$77% at comparable KL. (b) Accuracy drop vs. KL: imbalance (red diamonds) clusters near zero regardless of divergence magnitude. (c) Prior shift alone barely affects accuracy (97.4--97.7% across all imbalance levels). (d) Covariate shift causes exponential degradation ($\sigma{=}0.15 \to 77\%$). (e) Aggregated comparison confirms noise dominates. (f) Linear regression: $R^2 = 0.04$ shows aggregate KL is uninformative.
• Figure 4: Oracle samples are not memorized from training data. Distribution of nearest-neighbor distances (ResNet feature space) between generated and training images. The dashed line marks the memorization threshold ($d{=}10$); the bulk of distances lie well above, confirming sample novelty.
• Figure 5: Full scaling results on AFHQ. Epistemic uncertainty follows power-law decay across all architectures, with MobileNet showing the steepest decline and ViT the shallowest.