One-Shot Federated Ridge Regression: Exact Recovery via Sufficient Statistic Aggregation
Zahir Alsulaimawi
TL;DR
The paper addresses whether iterative communication is necessary for distributed ridge regression in federated settings. It introduces One-Shot σ-Fusion, where each client transmits sufficient statistics $G_k=A_k^ op A_k$ and $h_k=A_k^ op b_k$, enabling the server to compute the global solution $w_ = (G + abla I)^{-1} h$ in one shot, with $G= extstyle extstyle abla_k G_k$ and $h= extstyle extstyle abla_k h_k$. The authors prove exact recovery of the centralized ridge solution under mild coverage conditions, analyze privacy benefits of single-round noise addition, and explore practical extensions including random projections for high-dimensional data. Empirical results show one-shot fusion matches centralized accuracy while achieving up to 38× less communication; it remains robust to client dropout and supports federated cross-validation for hyperparameter selection. The framework extends to kernel methods via random features but is not applicable to general nonlinear architectures, offering a principled, efficient alternative to iterative federated optimization in appropriate linear or kernelized settings.
Abstract
Federated learning protocols require repeated synchronization between clients and a central server, with convergence rates depending on learning rates, data heterogeneity, and client sampling. This paper asks whether iterative communication is necessary for distributed linear regression. We show it is not. We formulate federated ridge regression as a distributed equilibrium problem where each client computes local sufficient statistics -- the Gram matrix and moment vector -- and transmits them once. The server reconstructs the global solution through a single matrix inversion. We prove exact recovery: under a coverage condition on client feature matrices, one-shot aggregation yields the centralized ridge solution, not an approximation. For heterogeneous distributions violating coverage, we derive non-asymptotic error bounds depending on spectral properties of the aggregated Gram matrix. Communication reduces from $\mathcal{O}(Rd)$ in iterative methods to $\mathcal{O}(d^2)$ total; for high-dimensional settings, we propose and experimentally validate random projection techniques reducing this to $\mathcal{O}(m^2)$ where $m \ll d$. We establish differential privacy guarantees where noise is injected once per client, eliminating the composition penalty that degrades privacy in multi-round protocols. We further address practical considerations including client dropout robustness, federated cross-validation for hyperparameter selection, and comparison with gradient-based alternatives. Comprehensive experiments on synthetic heterogeneous regression demonstrate that one-shot fusion matches FedAvg accuracy while requiring up to $38\times$ less communication. The framework applies to kernel methods and random feature models but not to general nonlinear architectures.