Safe Feature Elimination for Non-Negativity Constrained Convex Optimization

TL;DR

The paper addresses safe elimination of nonzero features in convex problems with non-negativity constraints by leveraging a primal-dual feasible pair and a duality-gap-based search set that contains the dual optimum. A dual line search is developed to construct a high-quality dual feasible point from a given primal iterate, enabling efficient application of SAFE with first-order methods. The authors prove that, under mild conditions and sufficient duality gap reduction, all zero features can be eliminated, and they demonstrate this framework by certifying the uniqueness of NNLS solutions in both synthetic and extremely ill-conditioned microscopy problems. The work provides practical algorithms for feature screening and contributes to reliable NNLS uniqueness certification, with potential to reduce computation in large-scale, nonnegative convex problems. It also opens avenues for extensions to other convex non-negativity constrained problems and to $1$-norm regularized settings.

Abstract

Inspired by recent work on safe feature elimination for $1$-norm regularized least-squares, we develop strategies to eliminate features from convex optimization problems with non-negativity constraints. Our strategy is safe in the sense that it will only remove features/coordinates from the problem when they are guaranteed to be zero at a solution. To perform feature elimination we use an accurate, but not optimal, primal-dual feasible pair, making our methods robust and able to be used on ill-conditioned problems. We supplement our feature elimination problem with a method to construct an accurate dual feasible point from an accurate primal feasible point; this allows us to use a first-order method to find an accurate primal feasible point, then use that point to construct an accurate dual feasible point and perform feature elimination. Under reasonable conditions, our feature elimination strategy will eventually eliminate all zero features from the problem. As an application of our methods we show how safe feature elimination can be used to robustly certify the uniqueness of non-negative least-squares (NNLS) problems. We give numerical examples on a well-conditioned synthetic NNLS problem and a on set of 40000 extremely ill-conditioned NNLS problems arising in a microscopy application.

Figures (4)

• Figure 1: Dual geometry of the feature elimination subproblem \ref{['eq:feat-elim-strong-concavity']}. The hyperplanes $\langle a_1,\nu\rangle = 0$ and $\langle a_2,\nu\rangle = 0$ are drawn, with the dual feasible set $\{\nu\,:\,A^T\nu\ge 0\}$ extending toward the upper right. The dual optimal point $\nu^\ast$ is guaranteed to be the search set $N$, which is a ball of radius $\sqrt{2L\epsilon}$ centered at $\hat{\nu}$. Since $\langle a_1,\nu\rangle > 0$ for all $\nu\in N$, the feature elimination subproblem \ref{['eq:feat-elim-strong-concavity']} has strictly positive optimal value and so feature $a_1$ can be eliminated. The figure is drawn such that $\langle a_2, \nu^\ast\rangle = 0$, so $a_2$ cannot be eliminated.
• Figure 2: Finding $\hat{\nu}$ from $\nu'$ and $\nu^\text{strict}$ via the dual line search \ref{['eq:dual-line-search']}.
• Figure 3: Pathological case for the continuity of the dual line search when $\nu^\text{strict}$ is on the boundary of the dual feasible set (i.e., $\nu^\text{strict}$ is not strictly dual feasible).
• Figure 4: Certifying uniqueness for a synthetic ${50\times 100}$ NNLS problem. The dashed line in the left figure shows the minimum number of eliminated features to certify uniqueness; the dotted line shows the maximum number of features that can be eliminated; the dash-dot line shows SAFE using the orthogonal projection \ref{['eq:orth-proj']} instead of the dual line search. In the middle figure, the dashed line shows the duality gap when $p^\ast>0$ is certified and Lemma \ref{['lem:glp-uniqueness']} can be invoked. The right figure shows the bound \ref{['eq:nnls-dist-bound']} on the distance from $\hat{x}$ to the optimal point $x^\ast$. The line labeled SAFE bound$^*$ uses $f(A\hat{x}) - f(Ax^\ast)$ in place of the duality gap $\epsilon = f(A\hat{x}) - g(\hat{\nu})$ in the bound \ref{['eq:nnls-dist-bound']} (i.e., uses only the first inequality of \ref{['eq:nnls-dist-bound']}). Though this requires knowledge of $x^\ast$, this shows that the slower convergence of the bound \ref{['eq:nnls-dist-bound']} (which we can compute without knowledge of $x^\ast$ or $\nu^\ast$) is due to the suboptimality of $\hat{\nu}$.

Theorems & Definitions (8)

• Lemma 4.1
• Lemma 4.2
• proof
• Theorem 4.3
• Theorem 5.1
• proof
• Corollary 5.2
• Lemma 6.1: Lemma 5 from slawski2013nonnegative