Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Line Search Method Based on Regression and Uncertainty Quantification

Sören Laue, Tomislav Prusina

TL;DR

The paper tackles inefficiencies in unconstrained optimization line searches, where traditional methods discard intermediate data while refining a step-length interval. It introduces a Bayesian optimization-based line search that models the 1D objective along the search ray with a cubic spline surrogate and a Gaussian uncertainty, selecting the next probe as the minimizer of the surrogate minus uncertainty and iterating until the strong Wolfe conditions are met. This approach uses all available information, including function values and gradients, and is designed to be simple to implement and integrate. On the CUTEst unconstrained benchmark, the method outperforms state-of-the-art line searches by solving more problems to optimality with equivalent resource usage, indicating improved data efficiency and solution quality for large-scale optimization tasks.

Abstract

Unconstrained optimization problems are typically solved using iterative methods, which often depend on line search techniques to determine optimal step lengths in each iteration. This paper introduces a novel line search approach. Traditional line search methods, aimed at determining optimal step lengths, often discard valuable data from the search process and focus on refining step length intervals. This paper proposes a more efficient method using Bayesian optimization, which utilizes all available data points, i.e., function values and gradients, to guide the search towards a potential global minimum. This new approach more effectively explores the search space, leading to better solution quality. It is also easy to implement and integrate into existing frameworks. Tested on the challenging CUTEst test set, it demonstrates superior performance compared to existing state-of-the-art methods, solving more problems to optimality with equivalent resource usage.

Efficient Line Search Method Based on Regression and Uncertainty Quantification

TL;DR

The paper tackles inefficiencies in unconstrained optimization line searches, where traditional methods discard intermediate data while refining a step-length interval. It introduces a Bayesian optimization-based line search that models the 1D objective along the search ray with a cubic spline surrogate and a Gaussian uncertainty, selecting the next probe as the minimizer of the surrogate minus uncertainty and iterating until the strong Wolfe conditions are met. This approach uses all available information, including function values and gradients, and is designed to be simple to implement and integrate. On the CUTEst unconstrained benchmark, the method outperforms state-of-the-art line searches by solving more problems to optimality with equivalent resource usage, indicating improved data efficiency and solution quality for large-scale optimization tasks.

Abstract

Unconstrained optimization problems are typically solved using iterative methods, which often depend on line search techniques to determine optimal step lengths in each iteration. This paper introduces a novel line search approach. Traditional line search methods, aimed at determining optimal step lengths, often discard valuable data from the search process and focus on refining step length intervals. This paper proposes a more efficient method using Bayesian optimization, which utilizes all available data points, i.e., function values and gradients, to guide the search towards a potential global minimum. This new approach more effectively explores the search space, leading to better solution quality. It is also easy to implement and integrate into existing frameworks. Tested on the challenging CUTEst test set, it demonstrates superior performance compared to existing state-of-the-art methods, solving more problems to optimality with equivalent resource usage.
Paper Structure (6 sections, 7 equations, 3 figures, 1 table)

This paper contains 6 sections, 7 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Line Search Method
  4. Experiments
  5. Conclusion
  6. Disclosure of Interests.

Figures (3)

  • Figure 1: The figures show three iterations of our proposed line search method. The blue line is the original function, the dashed red line is the cubic interpolation, and the dashed green line is the uncertainty. The x-axis shows the step length, while the y-axis shows the function value.
  • Figure 2: The figure compares our line search with the Moré-Thuente line search. In Subfigure \ref{['fig:badplot0']}, orange points indicate our method's query steps, while purple points indicate points tried by the Moré-Thuente line search. Dashed circles indicate the final step size of the respective method.It can be seen that our method obtains a step length with a considerably better function value. Subfigures \ref{['fig:badplot1']}- \ref{['fig:badplot4']} depict four iterations of our method. The blue line is the original function, the dashed red line is the cubic interpolation, and the dashed green line is the spline plus uncertainty.
  • Figure 3: The histogram of the relative running times of L-BFGS-B and GENO. Black lines represent minimum and maximum, while the red line represents the median. The x-axis represents the relative difference of the two methods computed as in Eqaution \ref{['eq:diff']}.