Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Privately Learning Decision Lists and a Differentially Private Winnow

Mark Bun, William Fang

TL;DR

The paper introduces differential privacy mechanisms for two core learning tasks: privately learning decision lists in the PAC model and privately learning large-margin halfspaces in the online model. It presents a DP-PAC learner for $\mathcal{F}$-decision lists based on a DP adaptation of Rivest’s iterative approach using the Exponential Mechanism, achieving near-optimal sample complexity with polynomial-time training. In the online setting, it develops DP-Winnow, a private analog of Winnow for $\rho$-margin halfspaces, combining ConfidentWinnow with the Sparse Vector Technique to obtain polylogarithmic dependence on the dimension and inverse polynomial dependence on the margin in the regret bound. The results extend to privately learning decision lists online via a margin-based reduction and include lower bounds showing fundamental limits for private online learning. Collectively, the work bridges classic learning theory with differential privacy, producing practical, privacy-preserving algorithms that match non-private guarantees in key regimes and illuminate the trade-offs in private online learning of expressive concept classes.

Abstract

We give new differentially private algorithms for the classic problems of learning decision lists and large-margin halfspaces in the PAC and online models. In the PAC model, we give a computationally efficient algorithm for learning decision lists with minimal sample overhead over the best non-private algorithms. In the online model, we give a private analog of the influential Winnow algorithm for learning halfspaces with mistake bound polylogarithmic in the dimension and inverse polynomial in the margin. As an application, we describe how to privately learn decision lists in the online model, qualitatively matching state-of-the art non-private guarantees.

Privately Learning Decision Lists and a Differentially Private Winnow

TL;DR

The paper introduces differential privacy mechanisms for two core learning tasks: privately learning decision lists in the PAC model and privately learning large-margin halfspaces in the online model. It presents a DP-PAC learner for -decision lists based on a DP adaptation of Rivest’s iterative approach using the Exponential Mechanism, achieving near-optimal sample complexity with polynomial-time training. In the online setting, it develops DP-Winnow, a private analog of Winnow for -margin halfspaces, combining ConfidentWinnow with the Sparse Vector Technique to obtain polylogarithmic dependence on the dimension and inverse polynomial dependence on the margin in the regret bound. The results extend to privately learning decision lists online via a margin-based reduction and include lower bounds showing fundamental limits for private online learning. Collectively, the work bridges classic learning theory with differential privacy, producing practical, privacy-preserving algorithms that match non-private guarantees in key regimes and illuminate the trade-offs in private online learning of expressive concept classes.

Abstract

We give new differentially private algorithms for the classic problems of learning decision lists and large-margin halfspaces in the PAC and online models. In the PAC model, we give a computationally efficient algorithm for learning decision lists with minimal sample overhead over the best non-private algorithms. In the online model, we give a private analog of the influential Winnow algorithm for learning halfspaces with mistake bound polylogarithmic in the dimension and inverse polynomial in the margin. As an application, we describe how to privately learn decision lists in the online model, qualitatively matching state-of-the art non-private guarantees.
Paper Structure (30 sections, 20 theorems, 37 equations)

This paper contains 30 sections, 20 theorems, 37 equations.

Table of Contents

  1. Introduction
  2. Privately PAC Learning Decision Lists
  3. Techniques.
  4. Private Online Learning of Halfspaces via Winnow
  5. Techniques.
  6. Related Work.
  7. Preliminaries
  8. Decision Lists.
  9. Large-Margin Halfspaces.
  10. Exponential Mechanism.
  11. Private PAC Learner for Decision Lists
  12. Algorithm Overview
  13. Algorithm Analysis
  14. Differentially Private Winnow for Online Learning Halfspaces
  15. ConfidentWinnow
  16. ...and 15 more sections

Key Result

theorem 1

Let $\mathcal{F}$ be a set of "features" and let $\mathcal{F}$-decision lists denote the class of decision lists where each conditional rule is a member of $\mathcal{F}$. Then there is an $(\varepsilon, \delta)$-differentially private learner for $\mathcal{F}$-decision lists using $\tilde{O}(|\mathc

Theorems & Definitions (33)

  • theorem 1: Informal Statement of Theorem \ref{['theorem: zCDP DP-GreedyCover']}
  • theorem 2: Informal Statement of Theorem \ref{['theorem: DP-Winnow privacy and regret']}
  • proposition 1
  • definition 1: DworkMNS06KasiviswanathanLNRS08
  • definition 2
  • proposition 2
  • proof
  • theorem 3
  • definition 3: Differentially Private Learning Against an Oblivious Adversary
  • lemma 1
  • ...and 23 more