Sampling-Free Privacy Accounting for Matrix Mechanisms under Random Allocation

TL;DR

This work addresses the challenge of providing deterministic privacy guarantees for matrix mechanisms under balls-in-bins random allocation, where Monte Carlo approaches yield bounds that hold only with high probability. The authors develop two sampling-free accountants: a Rényi divergence–based accountant that exploits dynamic programming to compute R_α exactly for p-banded strategy matrices (and bounds for non-banded cases), and a Conditional Composition Accountant that bounds the privacy profile via per-step dominating pairs and a controlled low-probability bad event, incorporating variational tail bounds to tighten guarantees. The Rényi approach yields a substantial speedup for DP-SGD accounting (from exponential to polynomial in key parameters) and extends to general subsampled matrix mechanisms, while the conditional composition method provides stronger guarantees in high-privacy regimes. Together, these methods offer deterministic, scalable privacy accounting with practical impact for training with correlated noise and random batching, enabling improved utility without sacrificing rigorous DP guarantees.

Abstract

We study privacy amplification for differentially private model training with matrix factorization under random allocation (also known as the balls-in-bins model). Recent work by Choquette-Choo et al. (2025) proposes a sampling-based Monte Carlo approach to compute amplification parameters in this setting. However, their guarantees either only hold with some high probability or require random abstention by the mechanism. Furthermore, the required number of samples for ensuring $(ε,δ)$-DP is inversely proportional to $δ$. In contrast, we develop sampling-free bounds based on Rényi divergence and conditional composition. The former is facilitated by a dynamic programming formulation to efficiently compute the bounds. The latter complements it by offering stronger privacy guarantees for small $ε$, where Rényi divergence bounds inherently lead to an over-approximation. Our framework applies to arbitrary banded and non-banded matrices. Through numerical comparisons, we demonstrate the efficacy of our approach across a broad range of matrix mechanisms used in research and practice.

Figures (6)

• Figure 1: The Gram matrix $\mathbf{G}$ for matrices of size $N=3000$ with $k=10$. For banded matrices $\mathbf{C}$, we consider Banded Matrix Factorization (BandMF) mckenna2024scaling and Banded Square Root Factorization (BSR) kalinin2024banded with bandwidth $p=64$. For banded inverse matrices, we consider Banded Inverse Matrix Factorization (BandInvMF) and Banded Inverse Square Root (BISR) kalinin2025back with bandwidths $p = 4$ and $p=64$ accordingly. We note that although the matrix $\mathbf{C}$ is banded, the matrix $\mathbf{G}$ is not; rather, it is cyclically banded, with the corner blocks filled by strictly positive entries. Moreover, we observe that banded inverse matrices show similar behavior: their Gram matrices have entries that decay rapidly outside the cyclic band.
• Figure 2: Comparison of the Rényi based and conditional composition based accountants with the Monte Carlo (MC) accountant for DP-SGD and Banded Square Root (BSR) at $\delta=10^{-5}$, with calibrated noise multipliers $\sigma$ shown as a function of the privacy budget $\varepsilon$. Each plot corresponds to a different choice of the number of iterations $n$, bandwidth $p$ and the number of epochs $k$.
• Figure 3: Comparison of the RDP based and conditional composition based accountants with the MC accountant for BISR, BandInvMF, BLT and BandMF at $\delta=10^{-5}$, with calibrated noise multipliers $\sigma$ shown as a function of the privacy budget $\varepsilon$. Each plot corresponds to a different choice of the number of iterations $n$, the number of epochs $k$, and parameter $p$.
• Figure 4: Comparison of the largest mixture component's posterior probability $\overline{\lambda_b}$ (smaller is better) between different allocations of failure probability for different factorizations and noise multipliers $\sigma$ under $\delta_E = 0.5e-5$ and the "remove" relation. Using a union bound for the first epoch and a max bound for subsequent epochs ("Per-epoch max ($e \geq 2$)) performs best across most steps. Also using a max bound for the first epoch ("Per-epoch max ($e\geq 1$)) is only marginally better in the last few steps of the first epoch. Using a max bound across all steps ("Max") is only marginally better in the last epoch and sacrifices the low subsampling rate $\overline{\lambda_b}$ of the first epoch's conditional composition dominating pairs.
• Figure 5: Sample histogram of the ternary privacy loss $L_{\tilde{P},\tilde{Q},\tilde{R}}$, whose lower tail is used to find reverse hazard upper bound $\overline{\lambda_b}$, shown at transitions between epochs. Samples are taken for DP-SGD with $k=4$ epochs, $b=100$ batches per epoch, and noise multiplier $\sigma=5$ for the "remove" relation. The histogram widens between epochs, which explains observed jumps in the reverse hazard bound.

Theorems & Definitions (48)

• Definition 2.1: Differential Privacy dwork2006differential
• Lemma 2.2: Proposition 2 from barthe2013beyond
• Definition 2.3
• Definition 2.4: zhu2022optimal
• Definition 2.5
• Definition 2.6
• Definition 2.7
• Lemma 2.7
• Theorem 3.1
• Lemma 3.1