Differentially Private Distributed Estimation and Learning

TL;DR

This work tackles privacy-preserving distributed estimation and learning in networks where agents observe private signals and share noisy estimates. It introduces two local DP mechanisms—Signal DP and Network DP—and derives optimal noise strategies, showing Laplace noise with scale $\Delta/\varepsilon$ (or network-adjusted scales) minimizes the cost of privacy and accelerates convergence. The authors provide finite-time convergence bounds for MVUE and online learning of expected values under both DP regimes, leveraging a doubly stochastic consensus framework with dynamic topologies. Empirical results on real-world datasets (GEM Households and US Power Grid) demonstrate that the proposed DP-enabled methods achieve strong privacy with minimal loss in convergence speed or estimation accuracy, outperforming existing first-order DP approaches. The work advances privacy-aware distributed decision-making in critical CPS contexts, offering online adaptability and heterogeneous budget extensions for practical deployments.

Abstract

We study distributed estimation and learning problems in a networked environment where agents exchange information to estimate unknown statistical properties of random variables from their privately observed samples. The agents can collectively estimate the unknown quantities by exchanging information about their private observations, but they also face privacy risks. Our novel algorithms extend the existing distributed estimation literature and enable the participating agents to estimate a complete sufficient statistic from private signals acquired offline or online over time and to preserve the privacy of their signals and network neighborhoods. This is achieved through linear aggregation schemes with adjusted randomization schemes that add noise to the exchanged estimates subject to differential privacy (DP) constraints, both in an offline and online manner. We provide convergence rate analysis and tight finite-time convergence bounds. We show that the noise that minimizes the convergence time to the best estimates is the Laplace noise, with parameters corresponding to each agent's sensitivity to their signal and network characteristics. Our algorithms are amenable to dynamic topologies and balancing privacy and accuracy trade-offs. Finally, to supplement and validate our theoretical results, we run experiments on real-world data from the US Power Grid Network and electric consumption data from German Households to estimate the average power consumption of power stations and households under all privacy regimes and show that our method outperforms existing first-order, privacy-aware, distributed optimization methods.

Figures (15)

• Figure 1: Two types of DP protections considered in this paper are signal DP, $\mathcal{M}^{S}$, and network DP, $\mathcal{M}^{N}$. The private signal of agent $i$ at round $t$ is denoted by $\bm s_{i, t}$, $\bm d_{i, t}$ corresponds to the noise added from agent $i$ at round $t$, and $\bm \nu_{i, t}$ corresponds to the estimate of agent $i$ at round $t$. Our theoretical guarantees delineate the relationship between communication resources ($t$ rounds), privacy budget (${\varepsilon}$-DP), and total error (TE). Signal and network DP imply different performance tradeoffs as detailed in \ref{['tab:detailed_bounds']}.
• Figure 1: Non-Private Distributed Minimum Variance Unbiased Estimation
• Figure 1: Sample Paths for MVUE and OL for the German Households Dataset with heterogeneous budgets (centralized solution). For the OL case, we plot the optimal privacy overhead $\sum_{i = 1}^n \frac{\bm \Delta_i}{\bm {\varepsilon}_i^\star}$ which we compare with the lower bound $\sum_{i = 1}^n \frac{\bm \Delta_i}{{\varepsilon}}$, and the upper bound $\sum_{i = 1}^n \frac{\bm \Delta_i}{{\varepsilon}_{i, \max}}$.
• Figure 2: Non-Private Online Learning of Expected Values
• Figure 2: Distribution of Daily Consumption (in kWh) for the GEM House openData with log-normal fits is shown on the left (for all measurements and Day 0 measurements), followed by a visualization of the generated random geometric network with $\rho = 0.1$. The next two figures show the US Power Network degree distribution with log-normal fit, followed by its visualization.

Theorems & Definitions (12)

• Theorem 3.1: Minimum Variance Unbiased Estimation with Signal DP
• Corollary 3.2: Total Error of Minimum Variance Unbiased Estimation with Network DP
• Theorem 4.1: Online Learning of Expected Values with Signal DP
• Theorem 4.2: Online Learning of Expected Values with Network DP
• proof
• proof
• proof
• Proposition C.1
• proof
• Proposition C.2