A Split-Client Approach to Second-Order Optimization

TL;DR

The paper introduces a split-client asynchronous framework for second-order optimization, decoupling gradient and curvature computations to overlap expensive Hessian processing with inexpensive gradient steps. It specializes the framework to cubic regularization, proving a provable $\sqrt{\tau}$ improvement in wall-clock time when curvature computation is delayed or inexact. The theoretical results quantify how initialization error, delay, and Hessian inexactness affect convergence, and the experiments show consistent wall-clock gains over vanilla cubic Newton and Lazy Hessian baselines across exact and approximate curvature. The work highlights practical efficiency gains from asynchronous curvature handling and suggests broad applicability to other second-order methods and distributed/hardware-heterogeneous setups.

Abstract

Second-order methods promise faster convergence but are rarely used in practice because Hessian computations and decompositions are far more expensive than gradients. We propose a \emph{split-client} framework where gradients and curvature are computed asynchronously by separate clients. This abstraction captures realistic delays and inexact Hessian updates while avoiding the manual tuning required by Lazy Hessian methods. Focusing on cubic regularization, we show that our approach retains strong convergence guarantees and achieves a provable wall-clock speedup of order $\sqrtτ$, where $τ$ is the relative time needed to compute and decompose the Hessian compared to a gradient step. Since $τ$ can be orders of magnitude larger than one in high-dimensional problems, this improvement is practically significant. Experiments on synthetic and real datasets confirm the theory: asynchronous curvature consistently outperforms vanilla and Lazy Hessian baselines, while maintaining second-order accuracy.

Figures (9)

• Figure 1: Compact comparison with relative compute time encoded by box width. (A) Vanilla recomputes and decomposes the Hessian each iteration (H slower than G; Dec slowest, causing blocking). (B) Lazy reuses decompositions between updates (still blocks on refresh and requires tuning). (C) Split-Client (ours) deliberately separates fast gradients from slower asynchronous curvature; a dashed arrow labeled "delay $\tau$" and extra spacing make the arrival lag explicit, yielding a $\sqrt{\tau}$ wall-clock speedup.
• Figure 2: Cubic Newton (exact curvature):Left—synthetic logistic regression; Right—a1a. Async (split-client) converges fastest in wall-clock; Lazy (tuned $p$) helps vs. Vanilla but remains slower than Async.
• Figure 3: Quasi-Newton + cubic (L-BFGS curvature):Left—synthetic; Right—a1a. Async dominates in wall-clock time; Lazy improves over Vanilla but is sensitive to $p$ and still trails Async.
• Figure 4: Time profiling vs. dimension: Decomposition rapidly becomes the dominant cost, while the cubic subproblem (once decomposed) is relatively cheap. This explains Async’s advantage: curvature work is overlapped rather than blocking.
• Figure 5: Objective value versus wall-clock time for smooth nonconvex regression. Async cubic converges significantly faster than both Lazy and Vanilla cubic methods.

Theorems & Definitions (1)

• Lemma B.1