First-order majorization-minimization meets high-order majorant: Boosted inexact high-order forward-backward method
Alireza Kabgani, Masoud Ahookhosh
TL;DR
To address structured nonsmooth and nonconvex problems of the form $\min_x \varphi(x)=f(x)+g(x)$ with $f$ smooth and potentially nonconvex and $g$ proper lsc, the paper develops a first-order majorization–minimization framework based on a high-order majorant of degree $p>1$. It shows that a function admits such a majorant iff it is $p$-paraconcave, deriving a high-order descent lemma and building the high-order forward-backward envelope (HiFBE) and mapping (HiFBS); it then introduces an inexact boosted HiFBA with line search and proves subsequential and global convergence, including linear rates under the Kurdyka–Łojasiewicz (KL) property. Theoretical results are complemented by practical algorithms and experiments on linear inverse problems and regularized nonnegative matrix factorization, demonstrating improved efficiency over standard subgradient and Bregman-based methods. The framework generalizes beyond Lipschitz or Hölder gradient continuity and provides a flexible toolkit for solving challenging nonconvex optimization problems with non-smooth regularizers.
Abstract
This paper introduces a first-order majorization-minimization framework based on a high-order majorant for continuous functions, incorporating a non-quadratic regularization term of degree $p>1$. Notably, it is shown to be valid if and only if the function is $p$-paraconcave, thus extending beyond Lipschitz and Hölder gradient continuity for $p \in (1,2]$, and implying concavity for $p>2$. In the smooth setting, this majorant recovers a variant of the classical descent lemma with quadratic regularization. Building on this foundation, we develop a high-order inexact forward-backward algorithm (HiFBA) and its line-search-accelerated variant, named Boosted HiFBA. For convergence analysis, we introduce a high-order forward-backward envelope (HiFBE), which serves as a Lyapunov function. We establish subsequential convergence under suitable inexactness conditions, and we prove global convergence with linear rates for functions satisfying the Kurdyka-Łojasiewicz inequality. Our preliminary experiments on linear inverse problems and regularized nonnegative matrix factorization highlight the efficiency of HiFBA and its boosted variant, demonstrating their potential for solving challenging nonconvex optimization problems.