WEEP: A Differentiable Nonconvex Sparse Regularizer via Weakly-Convex Envelope
Takanobu Furuhashi, Hidekata Hontani, Qibin Zhao, Tatsuya Yokota
TL;DR
Sparse regularization often relies on non-differentiable penalties that hinder gradient-based optimization. WEEP leverages the $1$-weakly convex envelope to convert a non-convex base penalty into a differentiable, $L$-smooth, and weakly-convex regularizer with a closed-form proximal operator, enabling seamless use with SGD, Adam, and L-BFGS. The paper derives the WEEP construction, provides closed-form proximal updates, and demonstrates superior performance on 1D compressive sensing with TV and real-world image denoising over conventional convex and non-convex baselines. This approach yields tunable sparsity with non-vanishing gradients, offering a practical, scalable tool for sparse recovery in signals and images and potential integration with deep learning pipelines.
Abstract
Sparse regularization is fundamental in signal processing and feature extraction but often relies on non-differentiable penalties, conflicting with gradient-based optimizers. We propose WEEP (Weakly-convex Envelope of Piecewise Penalty), a novel differentiable regularizer derived from the weakly-convex envelope framework. WEEP provides tunable, unbiased sparsity and a simple closed-form proximal operator, while maintaining full differentiability and L-smoothness, ensuring compatibility with both gradient-based and proximal algorithms. This resolves the tradeoff between statistical performance and computational tractability. We demonstrate superior performance compared to established convex and non-convex sparse regularizers on challenging compressive sensing and image denoising tasks.