Dual VC Dimension Obstructs Sample Compression by Embeddings

TL;DR

The paper investigates how much VC dimension must increase when embedding a VC class into an extremal class, showing that in general the increase is at least exponential in the original VC dimension and that a universal embedding achieving only linear or polynomial growth is impossible. It introduces the dual VC dimension and proves a sharp upper bound for extremal classes: vc*(C) ≤ 2 vc(C) + 1, contrasting with Assouad's general bound. A unified abstract-convexity framework is developed via Radon numbers, establishing that for extremal C the dual Radon number satisfies r(C) ≤ 2 vc(C) + 1 and linking to the topological Radon theorem through a cube complex Q(C). These results imply fundamental limitations of Floyd and Warmuth’s sample compression program and provide a topological route to understanding the trade-offs between VC, dual VC, and dual Radon parameters in extremal classes.

Abstract

This work studies embedding of arbitrary VC classes in well-behaved VC classes, focusing particularly on extremal classes. Our main result expresses an impossibility: such embeddings necessarily require a significant increase in dimension. In particular, we prove that for every $d$ there is a class with VC dimension $d$ that cannot be embedded in any extremal class of VC dimension smaller than exponential in $d$. In addition to its independent interest, this result has an important implication in learning theory, as it reveals a fundamental limitation of one of the most extensively studied approaches to tackling the long-standing sample compression conjecture. Concretely, the approach proposed by Floyd and Warmuth entails embedding any given VC class into an extremal class of a comparable dimension, and then applying an optimal sample compression scheme for extremal classes. However, our results imply that this strategy would in some cases result in a sample compression scheme at least exponentially larger than what is predicted by the sample compression conjecture. The above implications follow from a general result we prove: any extremal class with VC dimension $d$ has dual VC dimension at most $2d+1$. This bound is exponentially smaller than the classical bound $2^{d+1}-1$ of Assouad, which applies to general concept classes (and is known to be unimprovable for some classes). We in fact prove a stronger result, establishing that $2d+1$ upper bounds the dual Radon number of extremal classes. This theorem represents an abstraction of the classical Radon theorem for convex sets, extending its applicability to a wider combinatorial framework, without relying on the specifics of Euclidean convexity. The proof utilizes the topological method and is primarily based on variants of the Topological Radon Theorem.

Figures (3)

• Figure 1: $S'$ is a subsample of $S$ and $B$ is a binary string of additional information.
• Figure 2: A $2$-dimensional illustration of the cube complex for the extremal class $\mathcal{C} = \{000, 010, 110, 100, 001\}$. It has $5$ vertices ($0$-dimensional cubes), $5$ edges ($1$-dimensional cubes), and $1$ square ($2$-dimensional cube). The square corresponds to a cube $(Y, f)$ of $\mathcal{C}$ with $Y=\{1,2\}$ and $f\colon 3\mapsto 0$. A unique maximal edge, connecting $(0,0,0)$ and $(0,0,1)$, corresponds to a cube with $Y=\{3\}$ and $f\colon 1\mapsto 0, 2\mapsto 0$.
• Figure 3: A $2$-dimensional hyperplane arrangement consisting of $3$ lines and $7$ cells. The class $\mathcal{C}$ is maximum. Its VC dimension is two because $|\mathcal{C}|<8$. Its dual VC dimension is one because $|\mathcal{C}^\star|<4$. Its Radon number is three, because the three concept $+-+,-++,---$ are Radon indepedent.

Theorems & Definitions (45)

• Definition 1: VC Dimension
• Theorem 1: Sauer-Shelah-Perles Inequality sauer:72Shelah:72
• Definition 2: Maximum Classes
• Theorem 2: Pajor Inequality Pajor:1985
• Definition 3: Extremal Classes
• Definition 4
• Conjecture 1: floyd:95Warmuth:03
• Theorem A: Main Result I
• Definition 5: Dual Class and VC dimension
• Theorem 3: assouad:83