Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Private PAC Learning May be Harder than Online Learning

Mark Bun, Aloni Cohen, Rathin Desai

TL;DR

This work investigates the computational limits of privately PAC learning and its relation to online learning via Littlestone dimension. It constructs a leakage-based framework (EncThr/LEncThr) using a function-revealing encryption scheme with a leakage tfld that supports an efficient online learner but blocks polynomial-time private PAC learning under plausible cryptographic assumptions. The authors prove an explicit separation: LEncThr is efficiently online learnable with a polynomial mistake bound, yet there is no polynomial-time DP-PAC learner for LEncThr when tfld leakage is log-invariant. This establishes that polynomial-time online learnability and polynomial-time private PAC learnability are incomparable, informing the design and limits of privacy-preserving learning methods with cryptographic hardness implications.

Abstract

We continue the study of the computational complexity of differentially private PAC learning and how it is situated within the foundations of machine learning. A recent line of work uncovered a qualitative equivalence between the private PAC model and Littlestone's mistake-bounded model of online learning, in particular, showing that any concept class of Littlestone dimension $d$ can be privately PAC learned using $\mathrm{poly}(d)$ samples. This raises the natural question of whether there might be a generic conversion from online learners to private PAC learners that also preserves computational efficiency. We give a negative answer to this question under reasonable cryptographic assumptions (roughly, those from which it is possible to build indistinguishability obfuscation for all circuits). We exhibit a concept class that admits an online learner running in polynomial time with a polynomial mistake bound, but for which there is no computationally-efficient differentially private PAC learner. Our construction and analysis strengthens and generalizes that of Bun and Zhandry (TCC 2016-A), who established such a separation between private and non-private PAC learner.

Private PAC Learning May be Harder than Online Learning

TL;DR

This work investigates the computational limits of privately PAC learning and its relation to online learning via Littlestone dimension. It constructs a leakage-based framework (EncThr/LEncThr) using a function-revealing encryption scheme with a leakage tfld that supports an efficient online learner but blocks polynomial-time private PAC learning under plausible cryptographic assumptions. The authors prove an explicit separation: LEncThr is efficiently online learnable with a polynomial mistake bound, yet there is no polynomial-time DP-PAC learner for LEncThr when tfld leakage is log-invariant. This establishes that polynomial-time online learnability and polynomial-time private PAC learnability are incomparable, informing the design and limits of privacy-preserving learning methods with cryptographic hardness implications.

Abstract

We continue the study of the computational complexity of differentially private PAC learning and how it is situated within the foundations of machine learning. A recent line of work uncovered a qualitative equivalence between the private PAC model and Littlestone's mistake-bounded model of online learning, in particular, showing that any concept class of Littlestone dimension can be privately PAC learned using samples. This raises the natural question of whether there might be a generic conversion from online learners to private PAC learners that also preserves computational efficiency. We give a negative answer to this question under reasonable cryptographic assumptions (roughly, those from which it is possible to build indistinguishability obfuscation for all circuits). We exhibit a concept class that admits an online learner running in polynomial time with a polynomial mistake bound, but for which there is no computationally-efficient differentially private PAC learner. Our construction and analysis strengthens and generalizes that of Bun and Zhandry (TCC 2016-A), who established such a separation between private and non-private PAC learner.
Paper Structure (25 sections, 25 theorems, 31 equations, 1 figure, 3 algorithms)

This paper contains 25 sections, 25 theorems, 31 equations, 1 figure, 3 algorithms.

Table of Contents

  1. Introduction
  2. Techniques
  3. Preliminaries
  4. PAC Learning
  5. Differential Privacy
  6. Online Learning, Halving, and Thresholds
  7. Concept Class and its Learnability
  8. Computational Separation between PAC and Private PAC learning
  9. Function Revealing Encryption
  10. Correctness.
  11. On perfectly and strongly correct evaluation.
  12. Properties of leakage function for separation
  13. Sufficient Leakage for Online Learning
  14. Sufficient condition for Hardness of Private Learning
  15. Efficient Online Learner
  16. ...and 10 more sections

Key Result

theorem 2

Let $\mathcal{F}$ be a concept class with Littlestone dimension$d = L(\mathcal{F})$. Then $\tilde{O}(d^6)$ samples are sufficient to privately learn $\mathcal{F}$ and $\Omega(\log^* d)$ samples are necessary.

Figures (1)

  • Figure 1: Sketch of construction

Theorems & Definitions (46)

  • theorem 2: AlonBLMM22GhaziGKM21
  • theorem 3: Informal
  • definition 4
  • definition 5: PAC Learning, Valiant84
  • definition 6: Efficient PAC Learning
  • definition 7
  • definition 8: Differential Privacy, DworkMNS06DworkKMMN06
  • lemma 9: Basic Composition DworkMNS06DworkL09
  • lemma 10: Group Privacy
  • lemma 11: Post-Processing
  • ...and 36 more