Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Screening Cut Generation for Sparse Ridge Regression

Haozhe Tan, Guanyi Wang

TL;DR

A novel cut-generation method, Screening Cut Generation (SCG), is proposed, to eliminate non-optimal solutions for arbitrarily given samples and offers superior screening capability by identifying whether a specific combination of multiple features (binaries) lies in the set of optimal solutions.

Abstract

Sparse ridge regression is widely utilized in modern data analysis and machine learning. However, computing globally optimal solutions for sparse ridge regression is challenging, particularly when samples are arbitrarily given or generated under weak modeling assumptions. This paper proposes a novel cut-generation method, Screening Cut Generation (SCG), to eliminate non-optimal solutions for arbitrarily given samples. In contrast to recent safe variable screening approaches, SCG offers superior screening capability by identifying whether a specific $\{\pm 1\}$ combination of multiple features (binaries) lies in the set of optimal solutions. This identification is based on a convex relaxation solution rather than directly solving the original sparse ridge regression. Hence, the cuts generated by SCG can be applied in the pre-processing step of branch-and-bound and its variants to construct safe outer approximations of the optimal solution set. Numerical experiments are reported to validate the theoretical results and demonstrate the efficiency of SCG, particularly in hard real instances and synthetic instances with high dimensions, low ridge regularization parameters, or challenging modeling assumptions.

Screening Cut Generation for Sparse Ridge Regression

TL;DR

A novel cut-generation method, Screening Cut Generation (SCG), is proposed, to eliminate non-optimal solutions for arbitrarily given samples and offers superior screening capability by identifying whether a specific combination of multiple features (binaries) lies in the set of optimal solutions.

Abstract

Sparse ridge regression is widely utilized in modern data analysis and machine learning. However, computing globally optimal solutions for sparse ridge regression is challenging, particularly when samples are arbitrarily given or generated under weak modeling assumptions. This paper proposes a novel cut-generation method, Screening Cut Generation (SCG), to eliminate non-optimal solutions for arbitrarily given samples. In contrast to recent safe variable screening approaches, SCG offers superior screening capability by identifying whether a specific combination of multiple features (binaries) lies in the set of optimal solutions. This identification is based on a convex relaxation solution rather than directly solving the original sparse ridge regression. Hence, the cuts generated by SCG can be applied in the pre-processing step of branch-and-bound and its variants to construct safe outer approximations of the optimal solution set. Numerical experiments are reported to validate the theoretical results and demonstrate the efficiency of SCG, particularly in hard real instances and synthetic instances with high dimensions, low ridge regularization parameters, or challenging modeling assumptions.
Paper Structure (30 sections, 10 theorems, 34 equations, 2 figures, 9 tables, 2 algorithms)

This paper contains 30 sections, 10 theorems, 34 equations, 2 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Main Contributions & Paper Organization
  3. Notation
  4. Preliminary & Literature Review
  5. Main Results
  6. Screening Cut Generation Rule
  7. Minimal SCG-Tuple & Dominating Screening Cuts
  8. Connections With $k$-Cardinality Binary Knapsack Polytope
  9. Cuts With Strong Screening Ability
  10. SCG Algorithm Design
  11. Numerical Experiment
  12. Experimental Setup and Implementation
  13. Experimental Dataset
  14. Discussions on Numerical Results
  15. Factors That Influence SCG
  16. ...and 15 more sections

Key Result

Proposition 1

Denote the optimal solution of (eq:fenchel-relaxation) as $\hat{\bm{\beta}}$ and $\hat{\bm{p}}$. Assume there is no tie in element-wise square of $\hat{\bm{p}}$, i.e. $\hat{\bm{w}}:=\hat{\bm{p}} \circ \hat{\bm{p}}$. Let $v_{\tt ub}$ be any upper bound of $v^*$, usually obtained by any feasible solut identifies whether a binary $z_i$ could be zero or one in optimal solution sets with ${\tt gap} :=

Figures (2)

  • Figure 1: Total running time for synthetic datasets. The parameter configurations are specified in Section \ref{['sec:description-of-experimental-data']}. The first and second row corresponds to $\text{SNR} = 3.5$ and $\text{SNR} = 0.5$, respectively. The three columns, from left to right, are set as $d = 100$, $d = 300$, $d = 500$, respectively. Each dot represents the average value over $10$ independent datasets. The shaded area represents the inter-quartile range (25th to 75th percentiles).
  • Figure 2: Total running time for real datasets. The parameter configurations are specified in Section \ref{['sec:description-of-experimental-data']}. The four plots, arranged from left to right, represent the total running time under different $\gamma$ values with: whole CNAE dataset, reduced CNAE dataset with 200 samples, whole UJIndoorLoc dataset, reduced UJIndoorLoc dataset with 300 samples.

Theorems & Definitions (33)

  • Proposition 1: Restatement, Safe Screening Rules from Ata20
  • Definition 1: SCG-tuple $(S, N, C)$
  • Theorem 1: Screening Cuts Generation
  • Remark 1
  • Proposition 2
  • Definition 2
  • Proposition 3
  • Definition 3: Consecutive index set
  • Definition 4: Complementary index set
  • Proposition 4
  • ...and 23 more