An Adaptive Multiparameter Penalty Selection Method for Multiconstraint and Multiblock ADMM

TL;DR

The paper tackles penalty parameter selection in ADMM for multiconstraint and multiblock optimization by reframing ADMM as an affine fixed-point problem on the dual variable and aiming to minimize the spectral radius of the update operator. It introduces the multiparameter spectral radius approximation (MpSRA) rule, which adaptively updates individual penalty parameters using dual- and primal-variable differences, with practical safeguards and periodic updates. The approach extends prior single-parameter SRA theory to the multiparameter setting and accounts for complex eigenvalues, multiscaling covariance, and diagonal preconditioning equivalence. Across quadratic and nonquadratic (e.g., image reconstruction) experiments, MpSRA demonstrates robust convergence, resilience to problem scaling and initialization, and often faster runtimes compared with existing methods, including MpBBS and RB/SRB/SRA variants. The work offers a simple, implementable adaptive scheme with broad applicability to convex, multiconstraint ADMM problems in imaging and beyond.

Abstract

This work presents a new method for online selection of multiple penalty parameters for the alternating direction method of multipliers (ADMM) algorithm applied to optimization problems with multiple constraints or functionals with block matrix components. ADMM is widely used for solving constrained optimization problems in a variety of fields, including signal and image processing. Implementations of ADMM often utilize a single hyperparameter, referred to as the penalty parameter, which needs to be tuned to control the rate of convergence. However, in problems with multiple constraints, ADMM may demonstrate slow convergence regardless of penalty parameter selection due to scale differences between constraints. Accounting for scale differences between constraints to improve convergence in these cases requires introducing a penalty parameter for each constraint. The proposed method is able to adaptively account for differences in scale between constraints, providing robustness with respect to problem transformations and initial selection of penalty parameters. It is also simple to understand and implement. Our numerical experiments demonstrate that the proposed method performs favorably compared to a variety of existing penalty parameter selection methods.

Figures (8)

• Figure 1: Magnitude and angle (radians) of maximum eigenvalues of complex iteration matrix ${\bm{H}_{\boldsymbol{\rho}}}$ plotted as a function of $\rho_1$ and $\rho_2$. The maximum eigenvalues become real when either $\rho_1$ or $\rho_2$ are very large or very small. The the optimal fixed $\boldsymbol{\rho} = (\rho_1,\rho_2)^T$ corresponds to the location of the smallest magnitude, which occurs for a complex maximal eigenvalue.
• Figure 2: Magnitude and angle (radians) of maximum eigenvalues of complex iteration matrix ${\bm{H}_{\boldsymbol{\rho}}}$ plotted as a function of $\rho = \rho_1 = \rho_2$.
• Figure 3: Relative residual of ADMM solutions corresponding to an iteration matrix with complex eigenvalues for fixed penalty parameter, multiparameter BBS, and multiparameter SRA methods after 20 and 50 iterations plotted on a surface as a function of initial $\rho_1$ and $\rho_2$. Note that the structure of the fixed method's residual plots mimics the eigenvalue structure in Fig. \ref{['fig:eigen_example']} and only converges by 50 iterations for a specific set of $\boldsymbol{\rho}$. At 20 iterations the multiparameter SRA method has converged to a relative residual close to zero for the entire $\boldsymbol{\rho}$ search space, while the multiparameter BBS method requires 50 iterations.
• Figure 4: Relative residual of ADMM solutions corresponding to an iteration matrix with complex eigenvalues for single-parameter and multiparameter adaptive penalty parameter rules after 20 and 50 iterations plotted as a function of initial $\rho = \rho_1 = \rho_2$. Note that the structure of the fixed method's residual plots mimics the eigenvalue structure in Fig. \ref{['fig:diag_eigen_example']}. Both the multiparameter BBS and SRA method outperforms and converge quicker than the single-parameter methods, with the proposed multiparameter SRA method converging the quickest (20 iterations instead of 50).
• Figure 5: Relative residual after 50 iterations for ADMM solutions of $m=0$ (left), $m=1$ (middle), and $m=2$ (right) sum of quadratics problem for each adaptive penalty parameter method plotted as a function of initial $\rho = \rho_1 = \rho_2$. The optimal $\rho$ shifts for each case of $m$ and the single $\rho$ methods perform worse as the scaling between constraints grows. The multiparameter methods do not perform the best at the $m=0$ case when the constraints are scaled evenly. However, the performance of the multiparameter methods demonstrate stable performance as the scaling between constraints grows.

Theorems & Definitions (1)

• Definition 5.1: Multiscaling Covariant