Exploiting Agent Symmetries for Performance Analysis of Distributed Optimization Methods

TL;DR

This work develops a symmetry-driven framework for performance estimation in distributed optimization by embedding agent equivalence into the Performance Estimation Problem (PEP). By exploiting agent symmetries, the authors construct compact SDP formulations whose size depends on the number of equivalence classes $ r$ rather than the total number of agents $ n$, enabling worst-case analysis that is independent of $ n$ in many common settings. The approach is applied to the EXTRA algorithm to reveal worst-agent behavior, percentile performance, and robustness to local-function heterogeneity, producing sharper, scalable bounds and new insights into algorithm design. The results significantly advance automated performance analysis in distributed optimization, offering practical tools for tight bounds and informing parameter selection in large-scale networks.

Abstract

We show that, in many settings, the worst-case performance of a distributed optimization algorithm is independent of the number of agents in the system, and can thus be computed in the fundamental case with just two agents. This result relies on a novel approach that systematically exploits symmetries in worst-case performance computation, framed as Semidefinite Programming (SDP) via the Performance Estimation Problem (PEP) framework. Harnessing agent symmetries in the PEP yields compact problems whose size is independent of the number of agents in the system. When all agents are equivalent in the problem, we establish the explicit conditions under which the resulting worst-case performance is independent of the number of agents and is therefore equivalent to the basic case with two agents. Our compact PEP formulation also allows the consideration of multiple equivalence classes of agents, and its size only depends on the number of equivalence classes. This enables practical and automated performance analysis of distributed algorithms in numerous complex and realistic settings, such as the analysis of the worst agent performance. We leverage this new tool to analyze the performance of the EXTRA algorithm in advanced settings and its scalability with the number of agents, providing a tighter analysis and deeper understanding of the algorithm performance.

Figures (4)

• Figure 1: Comparison of the average functional error $E_f$\ref{['eq:perf_avg']} and the worst functional error $E_{f,\mathrm{worst}}$\ref{['eq:perf_worst']} for $t=15$ iterations of the EXTRA algorithm and their evolution with the number of agents $n$ in the system. The plot shows (i) the constant theoretical bound on $E_f$\ref{['eq:Ef_th']}, in black, (ii) the PEP bounds on the function error of the worst agent $E_{f,\mathrm{worst}}$, scaling sublinearly with $n$, in red and (iii) the PEP bounds for the function error of the average iterate $E_{f}$, in blue. The constant step-size $\alpha^*$ has been optimized for this performance criterion and leads to a bound constant with $n$ (plain blue line). Step-sizes can also be optimized with respect to $E_{f,\mathrm{worst}}(15)$, leading to smaller step-sizes $\alpha^*_{\mathrm{w}}(n)$ decreasing as $\frac{1}{\sqrt{n}}$. They improve the guarantee for $E_{f,\mathrm{worst}}$ and its scaling with $n$ but they deteriorate the corresponding guarantee for $E_{f}$ (see dashed lines). In this plot, the range of eigenvalues for the averaging matrix is $[-0.5,0.5]$ and the local functions are 1-smooth and 0.1-strongly convex.
• Figure 2: Comparison of the average iterates error $E_x$\ref{['eq:perf_avg']}, the worst iterates error $E_{x,\mathrm{worst}}$\ref{['eq:perf_worst']} and the 80-th percentile iterates error $E_{x,80}$\ref{['eq:PEP_80perc']} for $t=15$ iterations of the EXTRA algorithm and their evolution with the number of agents $n$ in the system. The plot shows (i) the PEP bounds on the iterates error of the worst agent $E_{x,\mathrm{worst}}$, scaling sublinearly with $n$, in red, (ii) the PEP bound on the 80-th percentile iterates error $E_{x,80}$, in green, and (iii) the PEP bounds for the average iterates error $E_{x}$, in blue. The constant step-size $\alpha^*$ has been hand-tuned for this performance criterion and leads to a bound constant with $n$ (plain blue line). Step-sizes can also be hand-tuned with respect to $E_{f\mathrm{worst}}(15)$, leading to smaller step-sizes $\alpha^*_{\mathrm{w}}(n)$. They improve the guarantee for $E_{f,\mathrm{worst}}$ and its scaling with $n$ without deteriorating too much the corresponding guarantee for $E_{f}$ (see dashed lines). In this plot, the range of eigenvalues for the averaging matrix is $[-0.5,0.5]$ and the local functions are 1-smooth and 0.1-strongly convex.
• Figure 3: Evolution with k of the k-th percentile of the agent performance for $t$ iterations of EXTRA, when the number of agents tends to infinity ($n \to \infty$). Different total numbers of iterations $t$ are compared. In this plot, the step-size is $\alpha = 0.78$, the range of eigenvalues for the averaging matrix is $[-0.5,0.5]$ and the local functions are 1-smooth and 0.1-strongly convex.
• Figure 4: Evolution with the number of agents $n$ of the worst-case average distance to optimum $E_{x}$ for 15 iterations of EXTRA with two equivalence classes of agents $\mathcal{V}_1$ and $\mathcal{V}_2$ using each a different condition number for their function class. The agents in $\mathcal{V}_1$ hold 1-smooth and 0.01-strongly convex local functions ($\kappa_1 = 100$) and the agents in $\mathcal{V}_2$ hold 1-smooth and 0.1-strongly convex local functions ($\kappa_2 = 10$). The guarantees are independent of the total number of agents $n$ and only depend on the proportion of agents in each class. In this experiment, the range of eigenvalues for the averaging matrix is $[-0.5,0.5]$.

Theorems & Definitions (87)

• Definition 1
• Definition 2: Class of distributed optimization methods $\mathcal{A}_{D}$
• Proposition 1: Interpolation constraints for $\mathcal{F}_{\mu,L}$ PEP_composite
• Definition 3: Gram-representable PEP_composite
• Proposition 2: PEP_composite
• Remark
• Definition 4: $\mathcal{W}_{\text{\footnotesize$(\lambda^-,\lambda^+)$}}$-interpolability
• Theorem 3
• Definition 5: $\mathcal{G}_{\text{\footnotesize$(\lambda^-,\lambda^+)$}}$
• Lemma 3.1