Efficient Differentiable Approximation of Generalized Low-rank Regularization

TL;DR

This work tackles the optimization bottleneck of low-rank regularization by introducing a differentiable, SVD-free approximation that covers a broad class of relaxations, including the nuclear norm and Schatten-$p$ norms. It defines a stochastic rank surrogate via random projections and leverages Taylor and Laguerre expansions to create differentiable estimators for generalized LRR, enabling gradient-based optimization on standard hardware. Convergence analysis shows that both bias and variance of the rank estimator decline as the sample size $N$ and iteration counts $k_1,k_2$ grow, supporting reliable optimization. Empirically, the method demonstrates versatility across matrix completion, video fore-background separation, and denoising with DNNs, delivering improved performance and efficient GPU-friendly computation. The accompanying code is available at the project GitHub repository.

Abstract

Low-rank regularization (LRR) has been widely applied in various machine learning tasks, but the associated optimization is challenging. Directly optimizing the rank function under constraints is NP-hard in general. To overcome this difficulty, various relaxations of the rank function were studied. However, optimization of these relaxed LRRs typically depends on singular value decomposition, which is a time-consuming and nondifferentiable operator that cannot be optimized with gradient-based techniques. To address these challenges, in this paper we propose an efficient differentiable approximation of the generalized LRR. The considered LRR form subsumes many popular choices like the nuclear norm, the Schatten-$p$ norm, and various nonconvex relaxations. Our method enables LRR terms to be appended to loss functions in a plug-and-play fashion, and the GPU-friendly operations enable efficient and convenient implementation. Furthermore, convergence analysis is presented, which rigorously shows that both the bias and the variance of our rank estimator rapidly reduce with increased sample size and iteration steps. In the experimental study, the proposed method is applied to various tasks, which demonstrates its versatility and efficiency. Code is available at https://github.com/naiqili/EDLRR.

Figures (9)

• Figure 1: Comparison of matrix completion for text removal. (a) ground truth; (b) image with text; (c)-(j) recovered images.
• Figure 2: The results using our algorithm in fore-background separation. From top to bottom: original frames of the video, separated backgrounds, and foreground objects.
• Figure 3: Results of applying low-rank regularization and the proposed differentiable approximation technique in the DnCNN denoising model, measured in PSNR.
• Figure 4: Approximating the Laplace relaxation function with Laguerre's expansion, with degree 8-10 polynomials. 1) Visualization of the standard degree 0-5 Laguerre polynomials; 2) Visualization of the approximation quality with degree 8-10 polynomials.'
• Figure 5: Numerical analysis on the synthetic dataset.

Theorems & Definitions (33)

• Proposition 1: Iterative matrix pseudo-inverse ben1966iterative
• Proposition 2: NS iteration for matrix square root
• Proposition 3: Equivalent definition of matrix rank wright2022high
• Proposition 4
• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Proposition 3: Equivalent definition of matrix rank wright2022high
• proof