Ramsey Theorems for Trees and a General 'Private Learning Implies Online Learning' Theorem

TL;DR

The paper extends the DP-online learning connection beyond binary classes by proving that DP-learnability implies online learnability for general classification tasks. It achieves this via Ramsey-type theorems for trees, introducing a robust notion of subset type and proving type-monochromatic subtrees for m-subsets, with tailored bounds for comparable and incomparable pairs. A central result shows that private learning forces a finite Littlestone dimension in partial multiclass settings, and hence online learnability, while also addressing infinite label spaces and highlighting open questions about the reverse direction. The methods merge tree-based Ramsey theory with an interior-point lower bound, illustrating that Littlestone dimension acts as a fundamental barrier to privacy and providing a deeper understanding of DP versus online learning across broad learning scenarios.

Abstract

This work continues to investigate the link between differentially private (DP) and online learning. Alon, Livni, Malliaris, and Moran (2019) showed that for binary concept classes, DP learnability of a given class implies that it has a finite Littlestone dimension (equivalently, that it is online learnable). Their proof relies on a model-theoretic result by Hodges (1997), which demonstrates that any binary concept class with a large Littlestone dimension contains a large subclass of thresholds. In a follow-up work, Jung, Kim, and Tewari (2020) extended this proof to multiclass PAC learning with a bounded number of labels. Unfortunately, Hodges's result does not apply in other natural settings such as multiclass PAC learning with an unbounded label space, and PAC learning of partial concept classes. This naturally raises the question of whether DP learnability continues to imply online learnability in more general scenarios: indeed, Alon, Hanneke, Holzman, and Moran (2021) explicitly leave it as an open question in the context of partial concept classes, and the same question is open in the general multiclass setting. In this work, we give a positive answer to these questions showing that for general classification tasks, DP learnability implies online learnability. Our proof reasons directly about Littlestone trees, without relying on thresholds. We achieve this by establishing several Ramsey-type theorems for trees, which might be of independent interest.

Figures (8)

• Figure 1: In (\ref{['subfig:subtree']}) the colored vertices denote a candidate $S$ for a subtree. According to our definition, the $S$ colored in red is the only legal subtree. The example in (\ref{['subfig:counterexample']}) shows a coloring of pairs (coloring left relations, right relations, and incomparable relations in different colors) for which no monochromatic subtree can exist.
• Figure 2: This example shows two triplets of vertices that admit the same relations of left/right descendant for each pair of vertices, for appropriate ordering of the vertices. However, consider coloring red all triplets of the form as in (\ref{['subfig:general_type_a']}) (the single vertex is to the right of the chain of size two), and coloring blue all triplets of the form as in (\ref{['subfig:general_type_b']}) (the single vertex is to the left of the chain of size two). Then within any tree (finite or infinite) with every triplet of these two kinds colored in this way, there is no monochromatic subtree of depth $2$, as these two patterns must appear in any such subtree.
• Figure 3: In each of the examples, the vertices colored red represent $A$, while the vertices circled in red are the ones in $\bar{A}\setminus A$. According to \ref{['def:type_subset']}, each example represents a subset of $3$ vertices of a different type.
• Figure 4: These three examples summarize the definitions of a subtree discussed in Section \ref{['sec:related_work']}, ordered by their restrictiveness. sub-Figure (\ref{['subfig:subtree_def_1']}) shows a valid subtree only according to the most relaxed definition, while sub-Figure (\ref{['subfig:subtree_def_3']}) shows a valid subtree according to the most restrictive definition (and thus according to all other definitions as well).
• Figure 5: Consider $\mathcal{X}$ as vertices of an infinite complete binary decision tree such that every internal vertex is labeled with a unique point $x\in\mathcal{X}$, and define a partial concept class ${\mathcal{H}_n\subset\{0,1,\star\}^\mathcal{X}}$ which consists of all the partial concepts that realize exactly one branch to depth $n$, and label every point $x\in\mathcal{X}$ outside of those $n$ vertices with $\star$. The Littlestone dimension of $\mathcal{H}_n$ is $n$, while the threshold dimension is $\leq 2$. Define $\mathcal{H}=\bigcup \mathcal{H}_n$, and it holds that $\mathsf{LD} \mathopen{}\left(\mathcal{H}\right)\mathclose{} =\infty$, while $\mathsf{TD} \mathopen{}\left(\mathcal{H}\right)\mathclose{} \leq 2$. This example can be easily modified to a multiclass $\mathcal{H}$ over an infinite label domain $\mathcal{Y}$: for each hypothesis $h$, instead of labeling off-branch examples with $\star$, label them by a label $y_h\in\mathcal{Y}$ unique to $h$.

Theorems & Definitions (59)

• Theorem A: Ramsey for pairs
• Remark 1: Quantitative Bounds for Pairs
• Definition 2: Closure
• Definition 3: Types
• Theorem B: Ramsey for $m$-subsets
• Remark 4: Optimal Number of Colors
• Remark 5: Infinite Case
• Theorem C: Ramsey for $m$-chains
• Remark 6: Tightness of the Bound
• Theorem D