A novel Information-Driven Strategy for Optimal Regression Assessment

TL;DR

This work tackles the challenge of certifying global regression optimality without access to the true data-generating function. It introduces the Information Teacher, a data-driven, mutual-information–based criterion that uses the residual $R=Y-\hat{f}(X)$ under additive noise models to detect whether $\hat{f}$ achieves the MMSE, with formal asymptotic and finite-sample guarantees and distribution-free applicability. The method provides a necessary-and-sufficient MI-based condition for global optimality, along with strong consistency and exponentially-fast detection, validated on synthetic data where ground truth is known and on real-world data where it is not. By leveraging a distribution-free MI estimator and a validation-set-based implementation, the Information Teacher offers a principled surrogate for ground-truth assessment, enabling reliable model selection, early stopping, and meta-learning in settings with limited or costly data.

Abstract

In Machine Learning (ML), a regression algorithm aims to minimize a loss function based on data. An assessment method in this context seeks to quantify the discrepancy between the optimal response for an input-output system and the estimate produced by a learned predictive model (the student). Evaluating the quality of a learned regressor remains challenging without access to the true data-generating mechanism, as no data-driven assessment method can ensure the achievability of global optimality. This work introduces the Information Teacher, a novel data-driven framework for evaluating regression algorithms with formal performance guarantees to assess global optimality. Our novel approach builds on estimating the Shannon mutual information (MI) between the input variables and the residuals and applies to a broad class of additive noise models. Through numerical experiments, we confirm that the Information Teacher is capable of detecting global optimality, which is aligned with the condition of zero estimation error with respect to the -- inaccessible, in practice -- true model, working as a surrogate measure of the ground truth assessment loss and offering a principled alternative to conventional empirical performance metrics.

Figures (3)

• Figure 1: Performance of favorable case (median and IQR intervals)
• Figure 2: Performance of unfavorable case (median and IQR intervals)
• Figure 3: Performance of MLP Regressor (median and IQR intervals)