No Validation, No Problem: Predicting Model Performance from a Single Gradient

TL;DR

This work introduces a validation-free predictor for model quality based on the Frobenius norm of the classifier-head gradient computed from a single forward-backward pass on detached features, $\|g\|_F$. The proxy strongly anti-correlates with accuracy and correlates with loss across ImageNet-1k, COCO detection/segmentation, and diffusion models, enabling near-oracle tail-window checkpoint selection with a small, universal configuration or with light architecture-specific tuning. The method extends to scale-normalized variants, detection heads, and diffusion heads, and remains lightweight enough to be integrated as a drop-in for early stopping and lightweight NAS pre-screening. Empirically, it achieves substantial compute savings with high recall of the best candidates, while also providing guidance for within-family ranking (e.g., ResNets) and cross-task applicability. Overall, the approach offers a practical, label-free mechanism to monitor and select model checkpoints, potentially reducing training costs and accelerating neural architecture search.

Abstract

We propose a validation-free checkpointing signal from a single forward-backward pass: the Frobenius norm of the classifier-head gradient on one detached-feature batch, ||g||_F = ||dL/dW||_F. Across ImageNet-1k CNNs and Transformers, this proxy is strongly negative with Top-1 and positive with loss. Selecting the checkpoint with the minimum head gradient in a short tail window closes most of the gap to the oracle (4.24% +/- 2.00% with a universal setup, about 1.12% with light per-family tuning). For practical deployment, a head-scale normalization is more stable within classic CNN families (e.g., ResNets), while a feature-scale normalization works well for Transformers and modern CNNs. The same one-batch probe also predicts COCO detection/segmentation mAP. In diffusion (UNet/DDPM on CIFAR-10), it tracks progress and enables near-oracle tail-window selection; it is positively correlated with same-distribution probe MSE and negatively with FID (lower is better), so it can be used as a lightweight, label-free monitor. Validation labels are never used beyond reporting. The probe adds much less than 0.1% of an epoch and works as a drop-in for validation-free checkpoint selection and early stopping.

Figures (10)

• Figure 1: Top-1 vs head-gradient norm ($\|g\|_F$) on seven architectures. Lower $\|g\|_F$ indicates higher separability and correlates strongly (negatively) with Top-1.
• Figure 2: Selector gap. Top-1(best) minus Top-1(at step of minimal $\|g\|_F$). Most gaps are $0$–$0.03$, demonstrating near-optimal checkpoint selection without validation data.
• Figure 3: Comparison. Head-gradient norm versus confidence across models for Top-1 and loss. $\|g\|_F$ generally outperforms or matches confidence; the two are complementary.
• Figure 4: CNN detectors: strong negative gradient-performance correlation. Classification head gradient norm vs. mAP for six CNN-based object detectors spanning diverse paradigms (two-stage, one-stage, anchor-free). Lower gradient norms reliably predict better detection performance (Pearson $r=-0.814$, $p=0.048^*$, $r^2=0.66$). This relationship suggests that gradient magnitude serves as an indicator of feature quality in convolutional architectures.
• Figure 5: Transformers: head-gradient proxy $\|\nabla_W \mathcal{L}\|_F / \|W\|_F$ vs. AP50 on COCO val. AP50 is the COCO metric “Average Precision at IoU $=0.50$,” i.e., category-averaged AP computed at a single IoU threshold of $0.50$, following the official COCO evaluation lin2014microsoft.