Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Algorithm to Solve Cardinality Constrained Quadratic Optimization Problem with an Application to the Best Subset Selection in Regression

Vikram Singh, Min Sun

TL;DR

This work tackles cardinality-constrained quadratic optimization (CCQO), a problem of selecting a sparse subset of variables to minimize a quadratic objective within a box constraint. It introduces IBB$^{+}$, a tailored interval branch-and-bound framework that branches only at 0, uses a tight lower bound via minimum-over-box computations, and applies CC-based deletions to guarantee global convergence with finite steps. The method is applied to Best Subset Selection (BSS) in regression, enhanced by feasibility sampling (SFS) and QR-based LB updates, and is shown to outperform standard BB and be competitive with MIO solvers on synthetic high-dimensional data up to $p=2000$. The results indicate that IBB$^{+}$ provides a scalable, exact approach to CCQO with practical performance advantages, and the framework can accommodate additional linear inequalities and acceleration strategies for further gains.

Abstract

A lot of problems, from fields like sparse signal processing, statistics, portfolio selection, and machine learning, can be formulated as a cardinality constraint optimization problem. The cardinality constraint gives the problem a discrete nature, making it computationally challenging to solve as the dimension of the problem increases. In this work, we present an algorithm to solve the cardinality constraint quadratic optimization problem using the framework of the interval branch-and-bound. Interval branch-and-bound is a popular approach for finding a globally optimal solution in the field of global optimization. The proposed method is capable of solving problems of a wide range of dimensions. In particular, we solve the classical best subset selection problem in regression and compare our algorithm against another branch-and-bound method and GUROBI's quadratic mixed integer solver. Numerical results show that the proposed algorithm outperforms the first and is competitive with the second solver.

An Algorithm to Solve Cardinality Constrained Quadratic Optimization Problem with an Application to the Best Subset Selection in Regression

TL;DR

This work tackles cardinality-constrained quadratic optimization (CCQO), a problem of selecting a sparse subset of variables to minimize a quadratic objective within a box constraint. It introduces IBB, a tailored interval branch-and-bound framework that branches only at 0, uses a tight lower bound via minimum-over-box computations, and applies CC-based deletions to guarantee global convergence with finite steps. The method is applied to Best Subset Selection (BSS) in regression, enhanced by feasibility sampling (SFS) and QR-based LB updates, and is shown to outperform standard BB and be competitive with MIO solvers on synthetic high-dimensional data up to . The results indicate that IBB provides a scalable, exact approach to CCQO with practical performance advantages, and the framework can accommodate additional linear inequalities and acceleration strategies for further gains.

Abstract

A lot of problems, from fields like sparse signal processing, statistics, portfolio selection, and machine learning, can be formulated as a cardinality constraint optimization problem. The cardinality constraint gives the problem a discrete nature, making it computationally challenging to solve as the dimension of the problem increases. In this work, we present an algorithm to solve the cardinality constraint quadratic optimization problem using the framework of the interval branch-and-bound. Interval branch-and-bound is a popular approach for finding a globally optimal solution in the field of global optimization. The proposed method is capable of solving problems of a wide range of dimensions. In particular, we solve the classical best subset selection problem in regression and compare our algorithm against another branch-and-bound method and GUROBI's quadratic mixed integer solver. Numerical results show that the proposed algorithm outperforms the first and is competitive with the second solver.

Paper Structure

This paper contains 19 sections, 3 theorems, 13 equations, 6 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Best subset selection in regression
  3. Branch and bound algorithm for \ref{['bss']}
  4. Mixed integer optimization formulation for \ref{['bss']}
  5. Interval branch and bound to solve \ref{['ccqo']}
  6. Branching
  7. Bounding
  8. Deletion
  9. Solving the Best Subset Selection using IBB$^{+}$
  10. Feasibility sampling
  11. Finding $lb \, f(\cdot)$ using QR decomposition
  12. IBB$^{+}$ versus BB
  13. Numerical results
  14. Choosing the initial box
  15. Test data setup
  16. ...and 4 more sections

Key Result

Theorem 3.1

IBB$^{+}$ reaches the optimal solution of ccqo in a finite number of iterations.

Figures (6)

  • Figure 1: IBB$^{+}$ tree for $p=5$ and $k=2$.
  • Figure 2: BB tree for $p=5$ and $k=2$.
  • Figure 3: Box plots of Relative Gap $\%$ for examples 1, 2, and 3 in small, medium, and large dimension regimes with 4 SNR values in OD case with $k\in \{5,10 \}$ for 1) IBB$^{+}$; 2) BB; 3) MIO.
  • Figure 4: Box plots of Relative Gap $\%$ for examples 1, 2, and 3 in small, medium, and large dimension regimes with 4 SNR values in UD case with $k\in \{5,10 \}$ for 1) IBB$^{+}$; 2) BB; 3) MIO.
  • Figure 5: Performance profiles of CPU time for examples 1, 2, and 3 in small, medium, and large dimension regimes with 4 SNR values in OD case and $k\in \{5,10 \}$.
  • ...and 1 more figures

Theorems & Definitions (10)

  • Theorem 3.1
  • proof
  • Proposition 3.1
  • proof
  • Proposition 4.1
  • proof
  • Remark 4.1
  • Example 1
  • Example 2
  • Example 3