Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Models

Yixiao Wang, Zishan Shao, Ting Jiang, Aditya Devarakonda

TL;DR

This paper tackles the computational bottleneck of elastic-net penalized GLMs by introducing Enhanced Cyclic Coordinate Descent (ECCD), which unrolls coordinate updates with a Taylor-based correction to stabilize large-block updates. By freezing the Hessian within blocks and using a first-order Taylor expansion to update gradients, ECCD achieves substantial per-epoch speedups without sacrificing convergence or accuracy. Theoretical analysis provides a Taylor-update error bound and complexity guarantees, while experiments show up to about 8x speedups over glmnet on pathwise fits across diverse datasets, with robust memory behavior thanks to an active-set cap. Overall, ECCD offers a practical, scalable pathwise solver that extends efficiently to other GLMs and distributions, enabling faster model selection in high-dimensional settings.

Abstract

We present a novel enhanced cyclic coordinate descent (ECCD) framework for solving generalized linear models with elastic net constraints that reduces training time in comparison to existing state-of-the-art methods. We redesign the CD method by performing a Taylor expansion around the current iterate to avoid nonlinear operations arising in the gradient computation. By introducing this approximation, we are able to unroll the vector recurrences occurring in the CD method and reformulate the resulting computations into more efficient batched computations. We show empirically that the recurrence can be unrolled by a tunable integer parameter, $s$, such that $s > 1$ yields performance improvements without affecting convergence, whereas $s = 1$ yields the original CD method. A key advantage of ECCD is that it avoids the convergence delay and numerical instability exhibited by block coordinate descent. Finally, we implement our proposed method in C++ using Eigen to accelerate linear algebra computations. Comparison of our method against existing state-of-the-art solvers shows consistent performance improvements of $3\times$ in average for regularization path variant on diverse benchmark datasets. Our implementation is available at https://github.com/Yixiao-Wang-Stats/ECCD.

Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Models

TL;DR

This paper tackles the computational bottleneck of elastic-net penalized GLMs by introducing Enhanced Cyclic Coordinate Descent (ECCD), which unrolls coordinate updates with a Taylor-based correction to stabilize large-block updates. By freezing the Hessian within blocks and using a first-order Taylor expansion to update gradients, ECCD achieves substantial per-epoch speedups without sacrificing convergence or accuracy. Theoretical analysis provides a Taylor-update error bound and complexity guarantees, while experiments show up to about 8x speedups over glmnet on pathwise fits across diverse datasets, with robust memory behavior thanks to an active-set cap. Overall, ECCD offers a practical, scalable pathwise solver that extends efficiently to other GLMs and distributions, enabling faster model selection in high-dimensional settings.

Abstract

We present a novel enhanced cyclic coordinate descent (ECCD) framework for solving generalized linear models with elastic net constraints that reduces training time in comparison to existing state-of-the-art methods. We redesign the CD method by performing a Taylor expansion around the current iterate to avoid nonlinear operations arising in the gradient computation. By introducing this approximation, we are able to unroll the vector recurrences occurring in the CD method and reformulate the resulting computations into more efficient batched computations. We show empirically that the recurrence can be unrolled by a tunable integer parameter, , such that yields performance improvements without affecting convergence, whereas yields the original CD method. A key advantage of ECCD is that it avoids the convergence delay and numerical instability exhibited by block coordinate descent. Finally, we implement our proposed method in C++ using Eigen to accelerate linear algebra computations. Comparison of our method against existing state-of-the-art solvers shows consistent performance improvements of in average for regularization path variant on diverse benchmark datasets. Our implementation is available at https://github.com/Yixiao-Wang-Stats/ECCD.
Paper Structure (53 sections, 3 theorems, 56 equations, 4 figures, 27 tables, 6 algorithms)

This paper contains 53 sections, 3 theorems, 56 equations, 4 figures, 27 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generalized Linear Models
  4. $s$-Step and Block Coordinate Methods
  5. Preliminary
  6. Generalized Linear Model
  7. Traditional Coordinate Descent for Elastic Net Regression
  8. Efficient Screening and Warm Start Strategy
  9. Methodology
  10. Block Coordinate Descent
  11. Enhanced Cyclic Coordinate Descent
  12. Theoretical Analysis
  13. Experiments
  14. Numerical Experiments
  15. Convergence on ECCD
  16. ...and 38 more sections

Key Result

Theorem 1

Let the true coordinate update at iteration $(k-1)s+\ell$ be given by $\Delta \beta_{j_{\ell}}^{\text{true}} := \tilde{\phi}_{j_{\ell}} / \tilde{\psi}_{j_{\ell}}$, and let $\widehat{\Delta \beta_{j_{\ell}}} := \phi_{j_{\ell}} / \psi_{j_{\ell}}$ be the corresponding approximate update computed using

Figures (4)

  • Figure 1: Coefficient trajectories $\beta_j$ for the diabetes dataset over 100 values of the regularization $\lambda$ (see Table \ref{['tab:datasets']} in Appendix for dataset dimensions). ECCD achieves identical solutions and induced sparsity patterns to GLMnet.
  • Figure 2: Results on the duke dataset with $\alpha = 0.5$ and $\epsilon = 0.01$. (a) Convergence of BCD exhibits numerical instability as block size $s$ increases. (b) Convergence of ECCD remains stable even for $s = 32$. The convergence threshold is set to 1e-10. (c) Breakdown of ECCD’s runtime components.
  • Figure 3: Comparison runtimes across six benchmark datasets ($\alpha = 0.5$). Speedups are normalized so that glmnet is 1.0×. ECCD ($s = 8$) consistently achieves the highest acceleration on both datasets.
  • Figure 4: ECCD Time Decomposition Plot (Duke)

Theorems & Definitions (6)

  • Theorem 1: Taylor-Approximate Update Error Bound
  • Theorem 2: Time Complexity Analysis
  • Theorem 3: Space Complexity Analysis
  • proof
  • proof
  • proof