Back to Square Roots: An Optimal Bound on the Matrix Factorization Error for Multi-Epoch Differentially Private SGD

TL;DR

This work addresses multi-epoch differential privacy in SGD by refining the Matrix Factorization Mechanism. It introduces Banded Inverse Square Root (BISR), which enforces a p-band on the inverse correlation matrix $C^{-1}$ to derive explicit, tight multi-epoch error bounds and to enable FFT-friendly computation. The authors prove a new lower bound that matches the BISR upper bound, establishing asymptotic optimality, and they also propose BandInvMF for memory-constrained settings, validated with experiments that show competitive or superior performance to existing methods. Overall, the approach provides both theoretical clarity and practical efficiency for privacy-preserving, multi-epoch training with matrix factorization.

Abstract

Matrix factorization mechanisms for differentially private training have emerged as a promising approach to improve model utility under privacy constraints. In practical settings, models are typically trained over multiple epochs, requiring matrix factorizations that account for repeated participation. Existing theoretical upper and lower bounds on multi-epoch factorization error leave a significant gap. In this work, we introduce a new explicit factorization method, Banded Inverse Square Root (BISR), which imposes a banded structure on the inverse correlation matrix. This factorization enables us to derive an explicit and tight characterization of the multi-epoch error. We further prove that BISR achieves asymptotically optimal error by matching the upper and lower bounds. Empirically, BISR performs on par with state-of-the-art factorization methods, while being simpler to implement, computationally efficient, and easier to analyze.

Figures (5)

• Figure 1: RMSE comparison for Banded Square Root (BSR) and Banded Inverse Square Root (BISR) methods across varying bandwidths ($p$). The results are shown for a fixed matrix size of $n=16384$ and a participation number of $k=8$. For BSR, the choice $p = b$ is numerically optimal, while for BISR a smaller bandwidth suffices to achieve an optimal value.
• Figure 2: RMSE across varying matrix sizes for different factorization methods under multiple optimizer settings and participation levels. We showed that also in practice, BISR performs on par with, or even better than, BSR and BLT. However, methods based on numerical optimization, such as BandMF, may achieve superior performance in certain regimes.
• Figure 3: RMSE across different factorizations and optimization parameters $\alpha, \beta$, with small bandwidth. The plot shows that BandInvMF and BISR can significantly reduce RMSE for a small bandwidth regime, justifying the use of banded inverse methods instead of banded factorizations.
• Figure 4: Accuracy results for CIFAR-10 and IMDB in the small bandwidth (low-memory) regime. For CIFAR-10, both amplified (left) and non-amplified (right) results show that inverse factorization methods, BISR and Band-Inv-MF, achieve significantly higher accuracy compared to Band-MF. Both plots correspond to $(9, 10^{-5})$-DP, with training performed using momentum $\beta = 0.9$ and weight decay $\alpha = 0.9999$, which we found to be optimal (see Tables \ref{['tab:hyperparameters']} and \ref{['tab:accuracies']} in the appendix). For IMDB, we report accuracy from fine-tuning under the same low-memory regime, comparing amplified and non-amplified settings, with training performed using momentum $\beta = 0.95$ and weight decay $\alpha = 0.99999$ (see Tables \ref{['tab:hyperparameters_imdb']} and \ref{['tab:imdb_accuracies']}).
• Figure 5: Convergence of Band-Inv-MF under different settings: for participation numbers $k = 4, 16$, with and without momentum ($\beta$) and weight decay ($\alpha$), across various matrix sizes (iterations). In general, we observe that 20 steps are sufficient for the procedure to converge.

Theorems & Definitions (40)

• Theorem 1
• Theorem 2: Analytic Gaussian Mechanism balle2018improving
• Theorem 3: General Multi-Participation Lower Bound
• proof : Proof Sketch
• Definition 1: Banded Inverse Square Root (BISR)
• Theorem 4: Approximation Error
• proof : Proof sketch
• Corollary 1: Optimized BISR Approximation Error
• Lemma 1: Inverse Square Root of the Matrix $A_{\alpha, \beta}$
• Lemma 2