Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences

Francisco Daunas, Iñaki Esnaola, Samir M. Perlaza, H. Vincent Poor

Abstract

The solution to empirical risk minimization with $f$-divergence regularization (ERM-$f$DR) is presented under mild conditions on $f$. Under such conditions, the optimal measure is shown to be unique. Examples of the solution for particular choices of the function $f$ are presented. Previously known solutions to common regularization choices are obtained by leveraging the flexibility of the family of $f$-divergences. These include the unique solutions to empirical risk minimization with relative entropy regularization (Type-I and Type-II). The analysis of the solution unveils the following properties of $f$-divergences when used in the ERM-$f$DR problem: $i\bigl)$ $f$-divergence regularization forces the support of the solution to coincide with the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; and $ii\bigl)$ any $f$-divergence regularization is equivalent to a different $f$-divergence regularization with an appropriate transformation of the empirical risk function.

Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences

Abstract

The solution to empirical risk minimization with -divergence regularization (ERM-DR) is presented under mild conditions on . Under such conditions, the optimal measure is shown to be unique. Examples of the solution for particular choices of the function are presented. Previously known solutions to common regularization choices are obtained by leveraging the flexibility of the family of -divergences. These include the unique solutions to empirical risk minimization with relative entropy regularization (Type-I and Type-II). The analysis of the solution unveils the following properties of -divergences when used in the ERM-DR problem: -divergence regularization forces the support of the solution to coincide with the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; and any -divergence regularization is equivalent to a different -divergence regularization with an appropriate transformation of the empirical risk function.
Paper Structure (16 sections, 6 theorems, 11 equations)

This paper contains 16 sections, 6 theorems, 11 equations.

Table of Contents

  1. Introduction
  2. Empirical Risk Minimization Problem
  3. The ERM with $f$-Divergence Regularization
  4. Preliminaries
  5. Problem Formulation
  6. Solution to the ERM-$f$DR
  7. Examples
  8. Kullback-Leibler Divergence
  9. Reverse Relative Entropy Divergence
  10. Jeffrey's Divergence
  11. Hellinger Divergence
  12. Jensen-Shannon Divergence
  13. $\chi^2$ Divergence
  14. Analysis of Regularization Factor
  15. Equivalence of the $f$-Regularization via Transformation of the Empirical Risk
  16. ...and 1 more sections

Key Result

Theorem 1

If the function $f$ in EqOp_f_ERMRERNormal is strictly convex, differentiable and there exists a $\beta$ in then the solution to the optimization problem in EqOp_f_ERMRERNormal, denoted by $P^{(Q, \lambda )}_{{\bm{\Theta}}|\boldsymbol{Z}=\bm{z}}\in \bigtriangleup_{Q}({\mathcal{M}})$, is unique, and for all ${\bm{\theta}} \in \mathop{\mathrm{supp}}\nolimits Q$ satisfies

Theorems & Definitions (11)

  • Definition 1: Expected Empirical Risk
  • Definition 2: $f$-divergence csiszar1967information
  • Theorem 1
  • Corollary 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Example 1
  • Example 2
  • Theorem 2
  • ...and 1 more