Differentially Private Relational Learning with Entity-level Privacy Guarantees

TL;DR

The paper tackles entity-level differential privacy for relational learning by identifying two core difficulties: high gradient sensitivity when nodes participate in many relations and the complex, coupled sampling in relational training. It develops an adaptive gradient clipping strategy that scales thresholds based on node occurrence and defines a tractable coupled-sampling amplification bound for a cardinality-dependent sampling subclass. Integrating these ideas, the authors present a DP-SGD variant tailored for relational data and demonstrate its practical utility by privately fine-tuning text encoders on text-attributed graphs, achieving favorable privacy-utility trade-offs. The work advances privacy guarantees for graph-structured learning and provides concrete methodology and empirical validation for real-world relational applications.

Abstract

Learning with relational and network-structured data is increasingly vital in sensitive domains where protecting the privacy of individual entities is paramount. Differential Privacy (DP) offers a principled approach for quantifying privacy risks, with DP-SGD emerging as a standard mechanism for private model training. However, directly applying DP-SGD to relational learning is challenging due to two key factors: (i) entities often participate in multiple relations, resulting in high and difficult-to-control sensitivity; and (ii) relational learning typically involves multi-stage, potentially coupled (interdependent) sampling procedures that make standard privacy amplification analyses inapplicable. This work presents a principled framework for relational learning with formal entity-level DP guarantees. We provide a rigorous sensitivity analysis and introduce an adaptive gradient clipping scheme that modulates clipping thresholds based on entity occurrence frequency. We also extend the privacy amplification results to a tractable subclass of coupled sampling, where the dependence arises only through sample sizes. These contributions lead to a tailored DP-SGD variant for relational data with provable privacy guarantees. Experiments on fine-tuning text encoders over text-attributed network-structured relational data demonstrate the strong utility-privacy trade-offs of our approach. Our code is available at https://github.com/Graph-COM/Node_DP.

Figures (5)

• Figure 1: The mini-batch sampling process in relational learning is a two-stage sampling that first samples edges from full edge list (positive sampling) and then construct negative edges based on the sampled positive edges (negative sampling). A node can occur in multiple positive and negative edges, leading to a high sensitivity. Note than the entire batch changes due to the removal of node 1.
• Figure 2: Comparison of per-iteration RDP bound $\varepsilon(\alpha)$ (left two figures) and DP composition bound $\varepsilon_{\mathrm{DP}}(T)$ over $T$ iterations (right two figures), under different capped node degree $K$. Our bound (adaptive clipping) refers bound Eq.\ref{['eq:rdp_amp']}. Our bound (standard clipping) uses similar amplification analysis but with standard clipping. Naive bound is the Gaussian RDP bound $\varepsilon(\alpha)=\alpha/\sigma^2$.
• Figure 3: Utility of different $\#$negative sample per positive $k_{\text{neg}}$, degree capping $K$, and noise multiplier $\sigma$ for zero-shot relation prediction. The legend AMAZ-sports means training on AMAZ-cloth and testing on AMAZ-sports, and similar for the legend AMAZ-cloth. In the rightmost figure, numbers along each line indicate the corresponding privacy parameter $\varepsilon_{\mathrm{DP}}$ at different noise levels.
• Figure 4: Comparison of per-iteration RDP bound $\varepsilon(\alpha)$ (first row) and DP composition bound $\varepsilon_{\mathrm{DP}}(T)$ over $T$ iterations (second figures), under different number of negative edges per positive $k_{\mathrm{neg}}$. Our bound (adaptive clipping) refers to the bound in Theorem \ref{['thm:amp_rdp']}. Our bound (standard clipping) refers to the similar amplification bound with standard clipping in place of adaptive clipping. Naive bound is simply the Gaussian mechanism RDP bound $\varepsilon(\alpha)=\alpha/\sigma^2$.
• Figure 5: Comparison of per-iteration RDP bound $\varepsilon(\alpha)$ (first row) and DP composition bound $\varepsilon_{\mathrm{DP}}(T)$ over $T$ iterations (second figures), under different noise level $\sigma$. Our bound (adaptive clipping) refers to the bound in Theorem \ref{['thm:amp_rdp']}. Our bound (standard clipping) refers to the similar amplification bound with standard clipping in place of adaptive clipping. Naive bound is simply the Gaussian mechanism RDP bound $\varepsilon(\alpha)=\alpha/\sigma^2$.

Theorems & Definitions (13)

• Definition 3.1
• Definition 3.2
• Definition 3.3
• Proposition 4.1: Local sensitivity of adaptive clipping
• Definition 4.1: Coupled Sampling
• Theorem 4.1
• Corollary 4.1
• proof
• Theorem B.1
• proof