A federated Kaczmarz algorithm

TL;DR

The paper presents FedRK, a federated variant of the randomized Kaczmarz algorithm for solving large linear systems partitioned across multiple clients. It establishes linear convergence guarantees and offers a new proof framework for RK, while extending the approach to sparse recovery via hard thresholding and to least-squares problems where a horizon is approached in the inconsistent setting. The authors demonstrate the method's practicality through synthetic experiments and a real-data application (prostate cancer dataset) with feature selection outcomes aligned with Lasso. Overall, FedRK provides a privacy-preserving, communication-efficient iterative projection method with potential for robust distributed linear solving and federated feature selection.

Abstract

In this paper, we propose a federated algorithm for solving large linear systems that is inspired by the classic randomized Kaczmarz algorithm. We provide convergence guarantees of the proposed method, and as a corollary of our analysis, we provide a new proof for the convergence of the classic randomized Kaczmarz method. We demonstrate experimentally the behavior of our method when applied to related problems. For underdetermined systems, we demonstrate that our algorithm can be used for sparse approximation. For inconsistent systems, we demonstrate that our algorithm converges to a horizon of the least squares solution. Finally, we apply our algorithm to real data and show that it is consistent with the selection of Lasso, while still offering the computational advantages of the Kaczmarz framework and thresholding-based algorithms in the federated setting.

Figures (5)

• Figure 1: Suppose that $C_1$ is a line intersecting the sphere at the green dots (left), and $C_2$ is a plane intersecting the sphere at the red (dashed) diametric circle. Then, the pink diametric (dashed) circles in the second sphere are the points we remove in the $x$-sphere, and the pink dots in the third sphere are the points we remove in the $y$-sphere mentioned in Theorem \ref{['maintheorem1']}. More precisely, a neighborhood of those points is removed.
• Figure 2: FedRK with clients running different numbers of local iterations
• Figure 3: The number of times each feature was selected in the $50$ trials when Algorithm \ref{['algorithm3']} with sparsity level $s=9$ was applied.
• Figure 4: This experiment applies Algorithm \ref{['algorithm1']} to the augmented system $A'x=b$ to solve the least squares problem, where $A'$ is obtained from $A$ by augmenting $n-256$ noisy columns. The result shows that we can shrink the convergence horizon by adding a suitable amount of noisy columns.
• Figure 5: feature selection via Algorithm \ref{['algorithm3']}

Theorems & Definitions (17)

• Theorem 1
• Theorem 2
• Corollary 1
• Theorem 3: Implicit function theorem
• Theorem 4
• proof
• Corollary 2
• proof
• Corollary 3
• proof