Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Similarity Measure Between Functions with Applications to Statistical Learning and Optimization

Chengpiao Huang, Kaizheng Wang

TL;DR

The paper introduces a unified measure of functional similarity, $(\varepsilon,\delta)$-closeness, based on how sub-optimality gaps translate between functions and their sub-level sets, providing an analytic tool beyond the sup-norm. It demonstrates that this closeness yields fast statistical rates in empirical risk minimization when the population objective is regular (e.g., strongly convex) and clarifies the role of the multiplicative factor $e^{\varepsilon}$ in achieving these rates. It further shows how the framework subsumes various notions of variation in non-stationary online optimization and connects to local Rademacher complexity bounds, offering a cohesive view of learning and optimization under non-stationarity. The results provide practical guidelines for interpreting function similarity in terms of learning guarantees and online performance, with explicit bounds and conditions that relate empirical to population losses and incorporate problem geometry via $\varepsilon$, $\delta$, and convexity/smoothness assumptions.

Abstract

In this note, we present a novel measure of similarity between two functions. It quantifies how the sub-optimality gaps of two functions convert to each other, and unifies several existing notions of functional similarity. We show that it has convenient operation rules, and illustrate its use in empirical risk minimization and non-stationary online optimization.

A Similarity Measure Between Functions with Applications to Statistical Learning and Optimization

TL;DR

The paper introduces a unified measure of functional similarity, -closeness, based on how sub-optimality gaps translate between functions and their sub-level sets, providing an analytic tool beyond the sup-norm. It demonstrates that this closeness yields fast statistical rates in empirical risk minimization when the population objective is regular (e.g., strongly convex) and clarifies the role of the multiplicative factor in achieving these rates. It further shows how the framework subsumes various notions of variation in non-stationary online optimization and connects to local Rademacher complexity bounds, offering a cohesive view of learning and optimization under non-stationarity. The results provide practical guidelines for interpreting function similarity in terms of learning guarantees and online performance, with explicit bounds and conditions that relate empirical to population losses and incorporate problem geometry via , , and convexity/smoothness assumptions.

Abstract

In this note, we present a novel measure of similarity between two functions. It quantifies how the sub-optimality gaps of two functions convert to each other, and unifies several existing notions of functional similarity. We show that it has convenient operation rules, and illustrate its use in empirical risk minimization and non-stationary online optimization.
Paper Structure (9 sections, 4 theorems, 29 equations)

This paper contains 9 sections, 4 theorems, 29 equations.

Table of Contents

  1. Introduction
  2. Notation.
  3. A Measure of Similarity Between Functions
  4. Applications to Statistical Learning and Optimization
  5. Application to Empirical Risk Minimization
  6. Application to Non-Stationary Online Optimization
  7. Application to Local Rademacher Complexity Bounds
  8. Case 1.
  9. Case 2.

Key Result

Lemma 2.1

For any lower bounded $h: \Omega \to \mathbb{R}$ and $t \in \mathbb{R}$, define the sub-level set Two lower bounded functions $f,g : \Omega \to \mathbb{R}$ are $(\varepsilon, \delta)$-close if and only if

Theorems & Definitions (10)

  • Definition 2.1: Closeness
  • Lemma 2.1: Sub-level set characterization
  • Example 2.1
  • Lemma 2.2
  • proof
  • Lemma 2.3
  • proof
  • Remark 3.1: The role of $\varepsilon$
  • Proposition 4.1
  • proof : Proof of \ref{['prop-closeness-local']}