Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Numerical optimization for the compatibility constant of the lasso

Kei Hirose

Abstract

The compatibility constant plays an important role in evaluating the prediction error of the lasso in high-dimensional settings. However, the computation of the compatibility constant is generally difficult because it is a complicated nonconvex optimization problem. In this study, we present a numerical approach to compute the compatibility constant when the support of true regression coefficients is given. We show that the optimization problem reduces to a quadratic programming (QP) once the signs of the nonzero coefficients are specified. In this case, the compatibility constant can be obtained by solving QPs for all possible sign combinations. We also formulate a mixed-integer QP (MIQP) approach that can be applied when the number of true nonzero coefficients is large. We investigate the finite-sample behavior of the compatibility constant for simulated data under various parameter settings and compare the prediction error with its theoretical upper bound. The behavior of the compatibility constant in finite samples is also investigated through a real data analysis.

Numerical optimization for the compatibility constant of the lasso

Abstract

The compatibility constant plays an important role in evaluating the prediction error of the lasso in high-dimensional settings. However, the computation of the compatibility constant is generally difficult because it is a complicated nonconvex optimization problem. In this study, we present a numerical approach to compute the compatibility constant when the support of true regression coefficients is given. We show that the optimization problem reduces to a quadratic programming (QP) once the signs of the nonzero coefficients are specified. In this case, the compatibility constant can be obtained by solving QPs for all possible sign combinations. We also formulate a mixed-integer QP (MIQP) approach that can be applied when the number of true nonzero coefficients is large. We investigate the finite-sample behavior of the compatibility constant for simulated data under various parameter settings and compare the prediction error with its theoretical upper bound. The behavior of the compatibility constant in finite samples is also investigated through a real data analysis.

Paper Structure

This paper contains 20 sections, 2 theorems, 42 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Numerical optimization for compatibility constant
  4. Quadratic Programming
  5. Mixed-Integer Quadratic Programming
  6. Monte Carlo Simulation
  7. Evaluation of compatibility constant via QP
  8. Evaluation of the prediction error bound at each step in the proof of the oracle inequality
  9. Computation of upper bound via MIQP
  10. Real data analysis
  11. Concluding remarks
  12. Compatibility constant under compound symmetry
  13. Derivation of the MSPE bound in Proposition \ref{['prop:lassoaccuracy']}
  14. Step 1: Basic inequality.
  15. Step 2: Dual norm
  16. ...and 5 more sections

Key Result

Proposition 1

Suppose that $\lambda$ satisfies $\lambda \ge 2\sigma \sqrt{2/n (1+\log \left( \frac{p}{\delta} \right))}$ with $\delta \in (0,1)$. Then, with probability at least $1-\delta$, we have In particular, plugging $\lambda = 2\sigma \sqrt{2/n (1+\log \left( \frac{p}{\delta} \right))}$ into the above equation, we have

Figures (8)

  • Figure 1: Compatibility constant $\phi$ for varying $(n, p, \rho)$. The red dashed line indicates the upper bound of compatibility constant for population covariance matrix, while the black dashed line indicates $\phi=0$.
  • Figure 2: Scaled MSPE and upper bound, ${\rm MSPE}_{\rm scaled}$ and ${\rm bound}_{\rm scaled}$, for varying $(n, p, \rho)$.
  • Figure 3: Ratio between the MSPE and the upper bound, $\text{MSPE}/\text{Bound}$, for varying $(n, p, \rho)$.
  • Figure 4: Ratio between the actual MSPE and its bound for each step of the proof of inequalities. The seven steps are denoted by B (basic inequality), Du (dual norm), M (max inequality), T (triangle inequality), Dr (dropping the negative term), C (compatibility inequality), and F (final bound).
  • Figure 5: Compatibility constant obtained by MIQP and QP (upper panel) and their ratio, $\phi_{\mathrm{MIQP}} / \phi_{\mathrm{QP}}$ (lower panel). In the upper panel, the gray shaded regions summarize the distribution of the compatibility constant obtained by QP: the dark gray band represents between the first and third quartiles, while the light gray bands show the overall ranges from the minimum to the maximum. The boxplots show the MIQP-based estimates for different warm-start sizes $K$ and time limits.
  • ...and 3 more figures

Theorems & Definitions (5)

  • Definition 1
  • Proposition 1
  • Remark 1
  • Proposition 2
  • proof