Parameter Choices for Sparse Multi-Parameter Regularization with the $\ell_1$ Norm
Qianru Liu, Rui Wang, Yuesheng Xu
TL;DR
The paper addresses selecting multiple regularization parameters $\lambda_j$ in a multi-parameter $\ell_1$-regularized model $\min\{\psi(u)+\sum_j \lambda_j\|B_j u\|_1\}$ to enforce sparsity under several transform matrices. It derives sparsity characterizations under each transform and introduces iterative schemes, with a fixed-point proximity algorithm (FPPA) to compute the regularized solution and auxiliary vectors in challenging nondifferentiable-fidelity, non-full-rank cases. The key contributions include theoretical sparsity conditions, two practical parameter-choice algorithms for differentiable and nondifferentiable fidelities, and a convergence-guaranteed FPPA framework, all validated through extensive simulations on block/separable, nonseparable, compound denoising, and Fused SVM problems. These results enable flexible, efficient sparsity-aware regularization applicable to large-scale inverse problems and learning tasks where multiple sparsity-inducing transforms are relevant.
Abstract
This paper introduces a multi-parameter regularization approach using the $\ell_1$ norm, designed to better adapt to complex data structures and problem characteristics while offering enhanced flexibility in promoting sparsity in regularized solutions. As data volumes grow, sparse representations of learned functions become critical for reducing computational costs during function operations. We investigate how the selection of multiple regularization parameters influences the sparsity of regularized solutions. Specifically, we characterize the relationship between these parameters and the sparsity of solutions under transform matrices, enabling the development of an iterative scheme for selecting parameters that achieve prescribed sparsity levels. Special attention is given to scenarios where the fidelity term is non-differentiable, and the transform matrix lacks full row rank. In such cases, the regularized solution, along with two auxiliary vectors arising in the sparsity characterization, are essential components of the multi-parameter selection strategy. To address this, we propose a fixed-point proximity algorithm that simultaneously determines these three vectors. This algorithm, combined with our sparsity characterization, forms the basis of a practical multi-parameter selection strategy. Numerical experiments demonstrate the effectiveness of the proposed approach, yielding regularized solutions with both predetermined sparsity levels and satisfactory approximation accuracy.