Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Theoretical Limits of Learning with Label Differential Privacy

Puning Zhao, Chuan Ma, Li Shen, Shaowei Wang, Rongfei Fan

TL;DR

The paper addresses the fundamental question of how much accuracy is attainable when learning with label differential privacy. By formulating minimax lower bounds via hypothesis testing and constructing matching algorithms, it reveals that relaxing DP to protect only labels yields substantial gains in the local model but only constant-factor gains in the central model. Across classification and regression under both bounded and heavy-tailed noise, it derives precise rates that depend on Hölder smoothness $\beta$, Tsybakov margin $\gamma$, dimension $d$, and privacy budget $\epsilon$, illustrating a clear divergence between local and central settings. The results have practical implications for privacy-preserving learning, showing when it is worth sacrificing feature privacy to boost accuracy and how to design efficient labeled DP mechanisms.

Abstract

Label differential privacy (DP) is designed for learning problems involving private labels and public features. While various methods have been proposed for learning under label DP, the theoretical limits remain largely unexplored. In this paper, we investigate the fundamental limits of learning with label DP in both local and central models for both classification and regression tasks, characterized by minimax convergence rates. We establish lower bounds by converting each task into a multiple hypothesis testing problem and bounding the test error. Additionally, we develop algorithms that yield matching upper bounds. Our results demonstrate that under label local DP (LDP), the risk has a significantly faster convergence rate than that under full LDP, i.e. protecting both features and labels, indicating the advantages of relaxing the DP definition to focus solely on labels. In contrast, under the label central DP (CDP), the risk is only reduced by a constant factor compared to full DP, indicating that the relaxation of CDP only has limited benefits on the performance.

On Theoretical Limits of Learning with Label Differential Privacy

TL;DR

The paper addresses the fundamental question of how much accuracy is attainable when learning with label differential privacy. By formulating minimax lower bounds via hypothesis testing and constructing matching algorithms, it reveals that relaxing DP to protect only labels yields substantial gains in the local model but only constant-factor gains in the central model. Across classification and regression under both bounded and heavy-tailed noise, it derives precise rates that depend on Hölder smoothness , Tsybakov margin , dimension , and privacy budget , illustrating a clear divergence between local and central settings. The results have practical implications for privacy-preserving learning, showing when it is worth sacrificing feature privacy to boost accuracy and how to design efficient labeled DP mechanisms.

Abstract

Label differential privacy (DP) is designed for learning problems involving private labels and public features. While various methods have been proposed for learning under label DP, the theoretical limits remain largely unexplored. In this paper, we investigate the fundamental limits of learning with label DP in both local and central models for both classification and regression tasks, characterized by minimax convergence rates. We establish lower bounds by converting each task into a multiple hypothesis testing problem and bounding the test error. Additionally, we develop algorithms that yield matching upper bounds. Our results demonstrate that under label local DP (LDP), the risk has a significantly faster convergence rate than that under full LDP, i.e. protecting both features and labels, indicating the advantages of relaxing the DP definition to focus solely on labels. In contrast, under the label central DP (CDP), the risk is only reduced by a constant factor compared to full DP, indicating that the relaxation of CDP only has limited benefits on the performance.

Paper Structure

This paper contains 33 sections, 19 theorems, 221 equations, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Label DP
  5. Risk of Classification and Regression
  6. Classification
  7. Local Label DP
  8. Central Label DP
  9. Regression with Bounded Noise
  10. Local Label DP
  11. Central Label DP
  12. Regression with Heavy-tailed Noise
  13. Local Label DP
  14. Central Label DP
  15. Comparison with Central DP on Both Features and Labels
  16. ...and 18 more sections

Key Result

Proposition 1

Let $\mathbf{z}_i=M(\mathbf{x}_i,y_i)$ for $i=1,\ldots,N$. If $\mathcal{A}$ is a function of $(\mathbf{x}_i, \mathbf{z}_i)$, $i=1,\ldots, N$, then $\mathcal{A}$ is $\epsilon$-label CDP.

Theorems & Definitions (33)

  • Definition 1
  • Definition 2
  • Definition 3
  • Proposition 1
  • Proposition 2
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Remark 1
  • ...and 23 more