Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Discrepancy: A Closer Look at the Theory of Distribution Shift

Robi Bhattacharjee, Nick Rittler, Kamalika Chaudhuri

TL;DR

This work takes a closer look at the theory of distribution shift for a classifier from a source to a target distribution, and adopts an Invariant-Risk-Minimization (IRM)-like assumption connecting the distributions, and characterize conditions under which data from a source distribution is sufficient for accurate classification of the target.

Abstract

Many machine learning models appear to deploy effortlessly under distribution shift, and perform well on a target distribution that is considerably different from the training distribution. Yet, learning theory of distribution shift bounds performance on the target distribution as a function of the discrepancy between the source and target, rarely guaranteeing high target accuracy. Motivated by this gap, this work takes a closer look at the theory of distribution shift for a classifier from a source to a target distribution. Instead of relying on the discrepancy, we adopt an Invariant-Risk-Minimization (IRM)-like assumption connecting the distributions, and characterize conditions under which data from a source distribution is sufficient for accurate classification of the target. When these conditions are not met, we show when only unlabeled data from the target is sufficient, and when labeled target data is needed. In all cases, we provide rigorous theoretical guarantees in the large sample regime.

Beyond Discrepancy: A Closer Look at the Theory of Distribution Shift

TL;DR

This work takes a closer look at the theory of distribution shift for a classifier from a source to a target distribution, and adopts an Invariant-Risk-Minimization (IRM)-like assumption connecting the distributions, and characterize conditions under which data from a source distribution is sufficient for accurate classification of the target.

Abstract

Many machine learning models appear to deploy effortlessly under distribution shift, and perform well on a target distribution that is considerably different from the training distribution. Yet, learning theory of distribution shift bounds performance on the target distribution as a function of the discrepancy between the source and target, rarely guaranteeing high target accuracy. Motivated by this gap, this work takes a closer look at the theory of distribution shift for a classifier from a source to a target distribution. Instead of relying on the discrepancy, we adopt an Invariant-Risk-Minimization (IRM)-like assumption connecting the distributions, and characterize conditions under which data from a source distribution is sufficient for accurate classification of the target. When these conditions are not met, we show when only unlabeled data from the target is sufficient, and when labeled target data is needed. In all cases, we provide rigorous theoretical guarantees in the large sample regime.
Paper Structure (49 sections, 28 theorems, 113 equations, 1 figure, 6 algorithms)

This paper contains 49 sections, 28 theorems, 113 equations, 1 figure, 6 algorithms.

Table of Contents

  1. Introduction
  2. An Illustrative Example
  3. Guarantees Beyond Discrepancy
  4. Related Work
  5. Preliminaries
  6. Problem Statement and Goal
  7. Feature Maps
  8. Nearest Neighbors
  9. Margin Conditions
  10. The Statistical IRM Assumption
  11. Desirable Properties of Feature Maps
  12. Stating the Statistical IRM Assumption
  13. The Statistical IRM Theorem
  14. The Distance Dimension of $\Phi$
  15. Direct Generalization from $\mathcal{D}_{s}$
  16. ...and 34 more sections

Key Result

Theorem 1

Suppose $\phi^*$ realizes the Statistical IRM assumption. Then for all $\epsilon, \delta > 0$, there exists $N$ such that for all $n \geq N$, with probability $\geq 1-\delta$ over $S \sim \mathcal{D}_{s}^n$,

Figures (1)

  • Figure 1: Examples of similar distribution shift problems with different data demands. Faded data points are sampled from the target distribution, while the bold points are selected from the source. In all cases, we wish to generalize to the target via the selection a feature map from $\Phi = \{\phi_x, \phi_y\}$, with $\phi_x$ and $\phi_y$ denoting projection onto the $x$ and $y$ axis, respectively.

Theorems & Definitions (85)

  • Example 1
  • Example 2
  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Theorem 1: Statistical IRM Theorem
  • Definition 5
  • Definition 6
  • Theorem 2
  • ...and 75 more