optHIM: Hybrid Iterative Methods for Continuous Optimization in PyTorch
Nikhil Sridhar, Sajiv Shah
TL;DR
optHIM presents an open-source PyTorch-based toolkit for unconstrained continuous optimization that unifies line-search and trust-region methods with full autograd support for function, gradient, and Hessian information. It evaluates eleven variants, including GD, Newton, BFGS, DFP, L-BFGS, and SR1, across line-search and trust-region paradigms, and exposes a YAML-driven configuration for reproducible experiments. Across diverse benchmark functions, the results reveal clear trade-offs: Newton offers rapid local convergence but high per-iteration cost, quasi-Newton methods like BFGS and LBFGS provide strong efficiency and robustness, and CG-based trust-region subproblems often outperform simple Cauchy steps while SR1 can be unstable on some landscapes. By integrating modular models, solvers, and hyperparameters with autograd and device-agnostic execution, optHIM enables transparent experimentation and education in continuous optimization.
Abstract
We introduce optHIM, an open-source library of continuous unconstrained optimization algorithms implemented in PyTorch for both CPU and GPU. By leveraging PyTorch's autograd, optHIM seamlessly integrates function, gradient, and Hessian information into flexible line-search and trust-region methods. We evaluate eleven state-of-the-art variants on benchmark problems spanning convex and non-convex landscapes. Through a suite of quantitative metrics and qualitative analyses, we demonstrate each method's strengths and trade-offs. optHIM aims to democratize advanced optimization by providing a transparent, extensible, and efficient framework for research and education.