Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empirical Risk Minimization with $f$-Divergence Regularization

Francisco Daunas, Iñaki Esnaola, Samir M. Perlaza, H. Vincent Poor

Abstract

In this paper, the solution to the empirical risk minimization problem with $f$-divergence regularization (ERM-$f$DR) is presented and conditions under which the solution also serves as the solution to the minimization of the expected empirical risk subject to an $f$-divergence constraint are established. The proposed approach extends applicability to a broader class of $f$-divergences than previously reported and yields theoretical results that recover previously known results. Additionally, the difference between the expected empirical risk of the ERM-$f$DR solution and that of its reference measure is characterized, providing insights into previously studied cases of $f$-divergences. A central contribution is the introduction of the normalization function, a mathematical object that is critical in both the dual formulation and practical computation of the ERM-$f$DR solution. This work presents an implicit characterization of the normalization function as a nonlinear ordinary differential equation (ODE), establishes its key properties, and subsequently leverages them to construct a numerical algorithm for approximating the normalization factor under mild assumptions. Further analysis demonstrates structural equivalences between ERM-$f$DR problems with different $f$-divergences via transformations of the empirical risk. Finally, the proposed algorithm is used to compute the training and test risks of ERM-$f$DR solutions under different $f$-divergence regularizers. This numerical example highlights the practical implications of choosing different functions $f$ in ERM-$f$DR problems.

Empirical Risk Minimization with $f$-Divergence Regularization

Abstract

In this paper, the solution to the empirical risk minimization problem with -divergence regularization (ERM-DR) is presented and conditions under which the solution also serves as the solution to the minimization of the expected empirical risk subject to an -divergence constraint are established. The proposed approach extends applicability to a broader class of -divergences than previously reported and yields theoretical results that recover previously known results. Additionally, the difference between the expected empirical risk of the ERM-DR solution and that of its reference measure is characterized, providing insights into previously studied cases of -divergences. A central contribution is the introduction of the normalization function, a mathematical object that is critical in both the dual formulation and practical computation of the ERM-DR solution. This work presents an implicit characterization of the normalization function as a nonlinear ordinary differential equation (ODE), establishes its key properties, and subsequently leverages them to construct a numerical algorithm for approximating the normalization factor under mild assumptions. Further analysis demonstrates structural equivalences between ERM-DR problems with different -divergences via transformations of the empirical risk. Finally, the proposed algorithm is used to compute the training and test risks of ERM-DR solutions under different -divergence regularizers. This numerical example highlights the practical implications of choosing different functions in ERM-DR problems.
Paper Structure (40 sections, 26 theorems, 57 equations, 6 figures, 1 algorithm)

This paper contains 40 sections, 26 theorems, 57 equations, 6 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Contributions
  4. Preliminaries
  5. Notation
  6. The Learning Problem
  7. The ERM-$f$DR Primal And Dual Problems
  8. The ERM-$f$DR Primal Problem
  9. Examples
  10. Relative Entropy
  11. Reverse Relative Entropy
  12. Jeffreys Divergence
  13. Jensen-Shannon Divergence
  14. Hellinger Divergence
  15. $\chi^2$ Divergence
  16. ...and 25 more sections

Key Result

Theorem 1

Under Assumptions assume:a and assume:b, the solution to the optimization problem in EqOp_f_ERMRERNormal, denoted $P^{(Q, \lambda )}_{{\bm{\Theta}}|\boldsymbol{Z}=\bm{z}}\in \bigtriangleup_{Q}({\mathcal{M}})$, is unique, and for all ${\bm{\theta}} \in \mathop{\mathrm{supp}}\nolimits Q$, where the functions $\dot{f}^{-1}$ and $\mathsf{L}_{\bm{z}}$ are respectively defined in EqDefInvDiffF and EqLx

Figures (6)

  • Figure 1: Representation of the empirical risk transformation from the $f$-divergence induced by $f(u)=-\log(u)+u-1$ and the $g$-divergence induced by $g(u)=u\log(u)-u+1$.
  • Figure 2: $28\times28$ Image of a handwritten 6 from MNIST dataset.
  • Figure 3: $28\times28$ Image of a handwritten 7 from MNIST dataset.
  • Figure 4: Average Training Error: average of the expected empirical risks $\mathsf{R}_{\bm{z}_1}(P^{(Q, \lambda )}_{{\bm{\Theta}}|\boldsymbol{Z}=\bm{z}_1})$, computed for four different $f$-divergences: relative entropy, reverse relative entropy, Jensen-Shannon divergence, and Hellinger divergence defined in \ref{['Eq_f_KL_DivEq']}, \ref{['Eq_f_KLr_DivEq']}, \ref{['Eq_f_JS_DivEq']}, and \ref{['Eq_f_Hell_DivEq']}, respectively. The results are the average of 100 random partitions of the training datasets.
  • Figure 5: Average Test Error: average of the expected empirical risks $\mathsf{R}_{\bm{z}_2}(P^{(Q, \lambda )}_{{\bm{\Theta}}|\boldsymbol{Z}=\bm{z}_1})$, computed for four different $f$-divergences: relative entropy, reverse relative entropy, Jensen-Shannon divergence, and Hellinger divergence defined in \ref{['Eq_f_KL_DivEq']}, \ref{['Eq_f_KLr_DivEq']}, \ref{['Eq_f_JS_DivEq']}, and \ref{['Eq_f_Hell_DivEq']}, respectively. The results are the average of 100 random partitions of the training and test datasets.
  • ...and 1 more figures

Theorems & Definitions (32)

  • Definition 1: $f$-divergence csiszar1967information
  • Definition 2: Separable Empirical Risk Function perlaza2024ERMRER
  • Theorem 1
  • Lemma 1
  • Definition 3: Legendre-Fenchel transform boyd2004convex
  • Definition 4: Normalization Function
  • Theorem 2
  • Lemma 2
  • Theorem 3
  • Theorem 4
  • ...and 22 more