Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Group-realizable multi-group learning by minimizing empirical risk

Navid Ardeshir, Samuel Deng, Daniel Hsu, Jingwen Liu

TL;DR

This work analyzes multi-group learning under a group-realizability assumption, showing that ERM over the group-realizable concept class $\\mathcal{C}_{\\mathcal{G},\\mathcal{H}}$ achieves strong, per-group excess risk bounds even when the subpopulation family $\\mathcal{G}$ is infinite but has finite VC dimension. It derives explicit sample complexity guarantees that scale with $d_{\\mathcal{G},\\mathcal{H}}$, $d_{\\mathcal{G}}$, and the minimal group probability $\\gamma$, recovering realizable single-group rates up to a $\\log(1/\\epsilon)$ factor and improving over agnostic baselines in the multi-group setting. However, computing a proper $c \in \\mathcal{C}_{\\mathcal{G},\\mathcal{H}}$ consistent with data is NP-hard, even with an oracle for finding group-wise consistent hypotheses. The authors thus advocate improper learning as a practical alternative, showing a Tosch-style online-to-batch method can yield a performant classifier in polynomial time under reasonable conditions, albeit not necessarily within $\\mathcal{C}_{\\mathcal{G},\\mathcal{H}}$. Overall, the paper highlights a notable separation between statistical efficiency and computational tractability in group-realizable multi-group learning and outlines directions for oracle-efficient algorithms and broader applicability of improper learning.

Abstract

The sample complexity of multi-group learning is shown to improve in the group-realizable setting over the agnostic setting, even when the family of groups is infinite so long as it has finite VC dimension. The improved sample complexity is obtained by empirical risk minimization over the class of group-realizable concepts, which itself could have infinite VC dimension. Implementing this approach is also shown to be computationally intractable, and an alternative approach is suggested based on improper learning.

Group-realizable multi-group learning by minimizing empirical risk

TL;DR

This work analyzes multi-group learning under a group-realizability assumption, showing that ERM over the group-realizable concept class achieves strong, per-group excess risk bounds even when the subpopulation family is infinite but has finite VC dimension. It derives explicit sample complexity guarantees that scale with , , and the minimal group probability , recovering realizable single-group rates up to a factor and improving over agnostic baselines in the multi-group setting. However, computing a proper consistent with data is NP-hard, even with an oracle for finding group-wise consistent hypotheses. The authors thus advocate improper learning as a practical alternative, showing a Tosch-style online-to-batch method can yield a performant classifier in polynomial time under reasonable conditions, albeit not necessarily within . Overall, the paper highlights a notable separation between statistical efficiency and computational tractability in group-realizable multi-group learning and outlines directions for oracle-efficient algorithms and broader applicability of improper learning.

Abstract

The sample complexity of multi-group learning is shown to improve in the group-realizable setting over the agnostic setting, even when the family of groups is infinite so long as it has finite VC dimension. The improved sample complexity is obtained by empirical risk minimization over the class of group-realizable concepts, which itself could have infinite VC dimension. Implementing this approach is also shown to be computationally intractable, and an alternative approach is suggested based on improper learning.
Paper Structure (11 sections, 3 theorems, 16 equations)

This paper contains 11 sections, 3 theorems, 16 equations.

Table of Contents

  1. Introduction
  2. Setting
  3. Sample complexity
  4. Computational complexity
  5. Overview of the reduction.
  6. Notation.
  7. Reduction.
  8. Efficiency of searching for a consistent hypothesis.
  9. Implications.
  10. Improper multi-group learning under group-realizability
  11. Discussion and future directions

Key Result

proposition 1

There exists $\mathcal{G} \subseteq \lbrace0,1\rbrace^{\mathcal{X}}$ and $\mathcal{H} \subseteq \lbrace-1,1\rbrace^{\mathcal{X}}$ on the same input domain $\mathcal{X}$, each with finite VC dimension, such that $\mathcal{C}_{\mathcal{G},\mathcal{H}}$ has infinite VC dimension.

Theorems & Definitions (7)

  • definition 1
  • definition 2
  • proposition 1
  • proof
  • theorem 1
  • lemma 1: *vapnik1971uniform
  • proof