Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Global Optimization: A Machine Learning Approach

Dimitris Bertsimas, Georgios Margaritis

TL;DR

Global optimization with black-box/implicit constraints is challenging. The authors extend OCTHaGOn by introducing multiple ML models (SVMs, GBMs, MLPs), adaptive sampling, robust optimization, and relaxation strategies to form a unified MIO approximation. They validate on 81 instances (77 MINLPLib), reporting improved feasibility and optimality in the majority and better or faster results than BARON in 11 cases. The approach broadens applicability to convex, non-convex, and data-driven or simulation-based constraints, highlighting a practical data-driven route for global optimization with general primitives.

Abstract

Many approaches for addressing Global Optimization problems typically rely on relaxations of nonlinear constraints over specific mathematical primitives. This is restricting in applications with constraints that are black-box, implicit or consist of more general primitives. Trying to address such limitations, Bertsimas and Ozturk (2023) proposed OCTHaGOn as a way of solving black-box global optimization problems by approximating the nonlinear constraints using hyperplane-based Decision-Trees and then using those trees to construct a unified mixed integer optimization (MIO) approximation of the original problem. We provide extensions to this approach, by (i) approximating the original problem using other MIO-representable ML models besides Decision Trees, such as Gradient Boosted Trees, Multi Layer Perceptrons and Suport Vector Machines, (ii) proposing adaptive sampling procedures for more accurate machine learning-based constraint approximations, (iii) utilizing robust optimization to account for the uncertainty of the sample-dependent training of the ML models, and (iv) leveraging a family of relaxations to address the infeasibilities of the final MIO approximation. We then test the enhanced framework in 81 Global Optimization instances. We show improvements in solution feasibility and optimality in the majority of instances. We also compare against BARON, showing improved optimality gaps or solution times in 11 instances.

Global Optimization: A Machine Learning Approach

TL;DR

Global optimization with black-box/implicit constraints is challenging. The authors extend OCTHaGOn by introducing multiple ML models (SVMs, GBMs, MLPs), adaptive sampling, robust optimization, and relaxation strategies to form a unified MIO approximation. They validate on 81 instances (77 MINLPLib), reporting improved feasibility and optimality in the majority and better or faster results than BARON in 11 cases. The approach broadens applicability to convex, non-convex, and data-driven or simulation-based constraints, highlighting a practical data-driven route for global optimization with general primitives.

Abstract

Many approaches for addressing Global Optimization problems typically rely on relaxations of nonlinear constraints over specific mathematical primitives. This is restricting in applications with constraints that are black-box, implicit or consist of more general primitives. Trying to address such limitations, Bertsimas and Ozturk (2023) proposed OCTHaGOn as a way of solving black-box global optimization problems by approximating the nonlinear constraints using hyperplane-based Decision-Trees and then using those trees to construct a unified mixed integer optimization (MIO) approximation of the original problem. We provide extensions to this approach, by (i) approximating the original problem using other MIO-representable ML models besides Decision Trees, such as Gradient Boosted Trees, Multi Layer Perceptrons and Suport Vector Machines, (ii) proposing adaptive sampling procedures for more accurate machine learning-based constraint approximations, (iii) utilizing robust optimization to account for the uncertainty of the sample-dependent training of the ML models, and (iv) leveraging a family of relaxations to address the infeasibilities of the final MIO approximation. We then test the enhanced framework in 81 Global Optimization instances. We show improvements in solution feasibility and optimality in the majority of instances. We also compare against BARON, showing improved optimality gaps or solution times in 11 instances.
Paper Structure (28 sections, 14 equations, 7 figures, 3 tables)

This paper contains 28 sections, 14 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Machine Learning in Optimization
  3. Contributions
  4. OCTHaGOn
  5. Algorithmic process
  6. Enhancements
  7. Enhancement 1: ML Model Training
  8. Enhancement 2: Sampling
  9. Enhancement 3: Relaxations
  10. Enhancement 4: Robustness
  11. An Illustrative example
  12. Standard Form Generation
  13. Sampling of Nonlinear constraints
  14. Model Training
  15. MIO representation
  16. ...and 13 more sections

Figures (7)

  • Figure 1: Feasible region and samples of the nonlinear constraints.
  • Figure 2: ML approximation of the nonlinear constraints.
  • Figure 3: OCT approximator of $g_2(\bm{x})\leq 0.$
  • Figure 4: MLP approximator of $g_1(\bm{x})\leq 0.$
  • Figure 5: MIO approximation of feasible space.
  • ...and 2 more figures