The ensmallen library for flexible numerical optimization

TL;DR

The paper addresses the need for a flexible optimization toolkit capable of handling a wide range of objective types beyond traditional ML-focused frameworks. It introduces ensmallen, a C++ template-based framework that supports general $f(x)$, differentiable $f(x)$, separable $f(x) = \sum_i f_i(x)$, constrained $f(x)$, and categorical objectives, with callback hooks for progress, early stopping, and debugging. A large library of pre-built optimizers (46) including quasi-Newton and SGD variants is provided, and new optimizers can be added by implementing an Optimize() method for a given objective interface. The framework relies on compile-time inference and template metaprogramming to achieve high performance while supporting multiple data types and ensuring compatibility via compile-time checks. Empirical results on linear and logistic regression show performance advantages over several general-purpose frameworks, and the BSD-licensed, open-source project is used by mlpack with thorough documentation and a technical report detailing implementation.

Abstract

We overview the ensmallen numerical optimization library, which provides a flexible C++ framework for mathematical optimization of user-supplied objective functions. Many types of objective functions are supported, including general, differentiable, separable, constrained, and categorical. A diverse set of pre-built optimizers is provided, including Quasi-Newton optimizers and many variants of Stochastic Gradient Descent. The underlying framework facilitates the implementation of new optimizers. Optimization of an objective function typically requires supplying only one or two C++ functions. Custom behavior can be easily specified via callback functions. Empirical comparisons show that ensmallen outperforms other frameworks while providing more functionality. The library is available at https://ensmallen.org and is distributed under the permissive BSD license.

Figures (1)

• Figure 1: Example implementation of an objective function class for linear regression and usage of the L-BFGS optimizer. The optimizer can be easily changed by replacing ens::L_BFGS with another optimizer, such as ens::GradientDescent, or ens::SA which implements simulated annealing kirkpatrick1983optimization.