Connections between convex optimization algorithms and subspace correction methods
Boou Jiang, Jongho Park, Jinchao Xu
TL;DR
This work presents a unified framework that exposes deep connections among subspace correction methods, alternating projection, operator splitting, and ADMM-type algorithms through convex duality. By introducing dualization, the authors show that many classical algorithms (e.g., von Neumann, Dykstra, Peaceman–Rachford, Douglas–Rachford, ADMM) are dualizations of subspace-correction schemes applied to appropriate dual formulations, enabling systematic construction of parallel and multi-block variants with convergence guarantees. The results offer a cohesive lens to transfer theories and techniques across algorithm classes, and they enable new, scalable algorithms for large-scale convex optimization problems including ROF imaging and multinomial logistic regression. The framework also motivates refined convergence analyses for subspace correction methods in the convex setting and paves the way for robust, parallelizable optimization tools suitable for modern architectures.
Abstract
We show that a broad range of convex optimization algorithms, including alternating projection, operator splitting, and multiplier methods, can be systematically derived from the framework of subspace correction methods via convex duality. To formalize this connection, we introduce the notion of dualization, a process that transforms an iterative method for the dual problem into an equivalent method for the primal problem. This concept establishes new connections across these algorithmic classes, encompassing both well-known and new methods. In particular, we show that classical algorithms such as the von Neumann, Dykstra, Peaceman--Rachford, and Douglas--Rachford methods can be interpreted as dualizations of subspace correction methods applied to appropriate dual formulations. Beyond unifying existing methods, our framework enables the systematic development of new algorithms for convex optimization. For instance, we derive parallel variants of alternating projection and operator splitting methods, as dualizations of parallel subspace correction methods, that are well-suited for large-scale problems on modern computing architectures and offer straightforward convergence guarantees. We also propose new alternating direction method of multipliers-type algorithms, derived as dualizations of certain operator splitting methods. These algorithms naturally ensure convergence even in the multi-block setting, where the conventional method does not guarantee convergence when applied to more than two blocks. This unified perspective not only facilitates algorithm design and the transfer of theoretical results but also opens new avenues for research and innovation in convex optimization.