Hybrid Coordinate Descent for Efficient Neural Network Learning Using Line Search and Gradient Descent

TL;DR

The paper addresses training efficiency for overparameterized two-layer ReLU networks by introducing a hybrid coordinate-descent method that alternates between line-search-based updates and gradient-guided coordinate updates via a Jacobi scheme, controlled by a gradient threshold $dw$. The approach updates first-layer weights in parallel with $\alpha=\frac{1}{n}$ and uses a per-coordinate rule to switch between gradient steps and line-search directions, with an explicit line-search procedure that adjusts $\epsilon$ from $dw$ to minimize the mean-squared-error loss. Empirical results on synthetic data show faster per-epoch convergence for the hybrid method across several network widths, but higher memory usage and longer wall-clock time per training cycle compared to gradient descent; larger $dw$ generally improves convergence speed and reduces runtime, highlighting a trade-off between update quality and computation. The findings suggest that with optimized line search and parallel implementation (e.g., in C++), the hybrid coordinate-descent approach could achieve competitive training efficiency while benefiting from parallelism and potentially faster loss reduction per epoch.

Abstract

This paper presents a novel coordinate descent algorithm leveraging a combination of one-directional line search and gradient information for parameter updates for a squared error loss function. Each parameter undergoes updates determined by either the line search or gradient method, contingent upon whether the modulus of the gradient of the loss with respect to that parameter surpasses a predefined threshold. Notably, a larger threshold value enhances algorithmic efficiency. Despite the potentially slower nature of the line search method relative to gradient descent, its parallelizability facilitates computational time reduction. Experimental validation conducted on a 2-layer Rectified Linear Unit network with synthetic data elucidates the impact of hyperparameters on convergence rates and computational efficiency.

Figures (5)

• Figure 1: The natural logarithm of the loss with respect to epoch for the training of 2-layer ReLU network using gradient descent and our hybrid coordinate descent with various numbers of weight ($m$) in the first layer and with $dw=0.5$.
• Figure 2: The natural logarithm with respect to the computational time (wall-clock time) for training of 2-layer ReLU network in 100 epochs using gradient descent and our hybrid coordinate descent with various numbers of weight ($m$) in the first layer and with $dw=0.5$.
• Figure 3: The natural logarithm of the loss with respect to epoch for the training of 2-layer ReLU network using gradient descent and our hybrid coordinate descent with various numbers of weight ($m$) in the first layer and with $dw=0.5$.
• Figure 4: The natural logarithm of the loss with respect to epoch for the training of 2-layer ReLU network using our hybrid coordinate descent with various numbers of weight ($m$) in the first layer and with different value of $dw$.
• Figure 5: The natural logarithm of the loss with respect to time for the training of 2-layer ReLU network using our hybrid coordinate descent with various numbers of weight ($m$) in the first layer and with different value of $dw$.