Perturbing the Derivative: Wild Refitting for Model-Free Evaluation of Machine Learning Models under Bregman Losses
Haichen Hu, David Simchi-Levi
TL;DR
The paper addresses the challenge of obtaining high-probability excess-risk bounds for complex, opaque models trained with empirical risk minimization under general Bregman losses, without relying on global function-class structure. It introduces Wild Refitting with Bregman Loss, a model-free procedure that perturbs derivatives rather than outputs, using a randomized Rademacher scheme to form wild responses and retrain a new predictor. The core contribution is a non-asymptotic, high-probability bound on the excess risk that depends on the wild optimism term and several controllable components, valid under fixed design and black-box access to the trainer. This framework enables principled model evaluation for deep neural networks and generative models, with potential extensions to random design and high-dimensional settings, and invites empirical validation on real-world opaque models.
Abstract
We study the excess risk evaluation of classical penalized empirical risk minimization (ERM) with Bregman losses. We show that by leveraging the idea of wild refitting, one can efficiently upper bound the excess risk through the so-called "wild optimism," without relying on the global structure of the underlying function class. This property makes our approach inherently model-free. Unlike conventional analysis, our framework operates with just one dataset and black-box access to the training procedure. The method involves randomized Rademacher symmetrization and constructing artificially modified outputs by perturbation in the derivative space with appropriate scaling, upon which we retrain a second predictor for excess risk estimation. We establish high-probability performance guarantee under the fixed design setting, demonstrating that wild refitting under Bregman losses, with an appropriately chosen wild noise scale, yields a valid upper bound on the excess risk. Thus, our work is promising for theoretically evaluating modern opaque ML models, such as deep neural networks and generative models, where the function class is too complex for classical learning theory and empirical process techniques.