Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Renet: Principled and Efficient Relaxation for the Elastic Net via Dynamic Objective Selection

Albert Dorador

TL;DR

Renet delivers a principled Relaxed Elastic Net by decoupling variable selection from coefficient estimation and introducing adaptive relaxation with a tunable parameter $\theta$, yielding reduced shrinkage bias while preserving convexity. The method enforces sign-consistency, uses a solver-agnostic two-stage procedure, and integrates a theoretical and empirical justification for synergy with the 1-SE rule, enabling ultra-sparse models without sacrificing predictive fidelity. Theoretical results establish stability, grouping, and bias-reduction properties, while extensive experiments across 20 datasets show Renet outperforms standard Elastic Net and competes with Adaptive Elastic Net in various challenging regimes, with competitive computation thanks to an adaptive backend. Practically, Renet offers a robust, scalable alternative for high-dimensional regression where multicollinearity and low SNR complicate model selection and estimation, and the 1-SE-driven sparsity can be achieved with significantly less bias.

Abstract

We introduce Renet, a principled generalization of the Relaxed Lasso to the Elastic Net family of estimators. While, on the one hand, $\ell_1$-regularization is a standard tool for variable selection in high-dimensional regimes and, on the other hand, the $\ell_2$ penalty provides stability and solution uniqueness through strict convexity, the standard Elastic Net nevertheless suffers from shrinkage bias that frequently yields suboptimal prediction accuracy. We propose to address this limitation through a framework called \textit{relaxation}. Existing relaxation implementations rely on naive linear interpolations of penalized and unpenalized solutions, which ignore the non-linear geometry that characterizes the entire regularization path and risk violating the Karush-Kuhn-Tucker conditions. Renet addresses these limitations by enforcing sign consistency through an adaptive relaxation procedure that dynamically dispatches between convex blending and efficient sub-path refitting. Furthermore, we identify and formalize a unique synergy between relaxation and the ``One-Standard-Error'' rule: relaxation serves as a robust debiasing mechanism, allowing practitioners to leverage the parsimony of the 1-SE rule without the traditional loss in predictive fidelity. Our theoretical framework incorporates automated stability safeguards for ultra-high dimensional regimes and is supported by a comprehensive benchmarking suite across 20 synthetic and real-world datasets, demonstrating that Renet consistently outperforms the standard Elastic Net and provides a more robust alternative to the Adaptive Elastic Net in high-dimensional, low signal-to-noise ratio and high-multicollinearity regimes. By leveraging an adaptive solver backend, Renet delivers these statistical gains while offering a computational profile that remains competitive with state-of-the-art coordinate descent implementations.

Renet: Principled and Efficient Relaxation for the Elastic Net via Dynamic Objective Selection

TL;DR

Renet delivers a principled Relaxed Elastic Net by decoupling variable selection from coefficient estimation and introducing adaptive relaxation with a tunable parameter , yielding reduced shrinkage bias while preserving convexity. The method enforces sign-consistency, uses a solver-agnostic two-stage procedure, and integrates a theoretical and empirical justification for synergy with the 1-SE rule, enabling ultra-sparse models without sacrificing predictive fidelity. Theoretical results establish stability, grouping, and bias-reduction properties, while extensive experiments across 20 datasets show Renet outperforms standard Elastic Net and competes with Adaptive Elastic Net in various challenging regimes, with competitive computation thanks to an adaptive backend. Practically, Renet offers a robust, scalable alternative for high-dimensional regression where multicollinearity and low SNR complicate model selection and estimation, and the 1-SE-driven sparsity can be achieved with significantly less bias.

Abstract

We introduce Renet, a principled generalization of the Relaxed Lasso to the Elastic Net family of estimators. While, on the one hand, -regularization is a standard tool for variable selection in high-dimensional regimes and, on the other hand, the penalty provides stability and solution uniqueness through strict convexity, the standard Elastic Net nevertheless suffers from shrinkage bias that frequently yields suboptimal prediction accuracy. We propose to address this limitation through a framework called \textit{relaxation}. Existing relaxation implementations rely on naive linear interpolations of penalized and unpenalized solutions, which ignore the non-linear geometry that characterizes the entire regularization path and risk violating the Karush-Kuhn-Tucker conditions. Renet addresses these limitations by enforcing sign consistency through an adaptive relaxation procedure that dynamically dispatches between convex blending and efficient sub-path refitting. Furthermore, we identify and formalize a unique synergy between relaxation and the ``One-Standard-Error'' rule: relaxation serves as a robust debiasing mechanism, allowing practitioners to leverage the parsimony of the 1-SE rule without the traditional loss in predictive fidelity. Our theoretical framework incorporates automated stability safeguards for ultra-high dimensional regimes and is supported by a comprehensive benchmarking suite across 20 synthetic and real-world datasets, demonstrating that Renet consistently outperforms the standard Elastic Net and provides a more robust alternative to the Adaptive Elastic Net in high-dimensional, low signal-to-noise ratio and high-multicollinearity regimes. By leveraging an adaptive solver backend, Renet delivers these statistical gains while offering a computational profile that remains competitive with state-of-the-art coordinate descent implementations.
Paper Structure (23 sections, 5 theorems, 17 equations, 1 figure, 6 tables, 2 algorithms)

This paper contains 23 sections, 5 theorems, 17 equations, 1 figure, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Motivation
  3. The Relaxation Problem
  4. Our Contribution: Renet
  5. The Renet Algorithm
  6. Stage 1: Feature Selection
  7. Stage 2: Adaptive Relaxation
  8. Solver-Agnosticism
  9. Computational Advantage over LARS-based Implementations
  10. Stability Safeguards and the Consensus on the $n \ll p$ Regime
  11. Theoretical Properties
  12. Generalization of Relaxed Lasso and Elastic Net
  13. Renet as a Stabilized Lasso
  14. The Renet Grouping Property
  15. Bias Reduction in Orthogonal Design
  16. ...and 8 more sections

Key Result

Proposition 1

Given a fixed hyperparameter triplet $(\lambda, \alpha, \theta)$ and an active set $\mathcal{A}_{\lambda}$, the Renet estimates $\hat{\beta}_{Renet}$ are the minimizers of:

Figures (1)

  • Figure 1: Performance is visualized as the absolute change relative to scikit-learn's Elastic Net (centered at 0,0). Ellipses show a 1-SE confidence region of the bivariate mean, pooled across datasets, folds, and seeds.

Theorems & Definitions (11)

  • Definition 1: Renet Estimator
  • Proposition 1: Lasso Stabilization
  • proof
  • Proposition 2: Renet Grouping Effect
  • proof
  • Proposition 3: Bias Mitigation Factor
  • proof
  • Corollary 1: Signal Recovery Ratio
  • proof
  • Proposition 4: 1-SE Renet Dominance Hierarchy
  • ...and 1 more