Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

High-dimensional Optimization with Low Rank Tensor Sampling and Local Search

Konstantin Sozykin, Andrei Chertkov, Anh-Huy Phan, Ivan Oseledets, Gleb Ryzhakov

TL;DR

TESALOCS tackles $d$-dimensional, non-convex optimization by fusing a TT-format low-rank discrete surrogate for global exploration with a local optimizer for refinement, updating the TT cores via SGD on a log-likelihood loss. The method achieves scalable exploration with memory that grows linearly in $d$, and it demonstrates improvements over a suite of gradient-based and gradient-free baselines under a fixed budget $M$. Through 20 non-convex, 100-dimensional benchmarks, TESALOCS attains orders-of-magnitude better results, illustrating its effectiveness for challenging high-dimensional problems. The work offers a practical, memory-efficient framework for global-to-local optimization with potential impacts on hyperparameter tuning and materials/design applications.

Abstract

We present a novel method called TESALOCS (TEnsor SAmpling and LOCal Search) for multidimensional optimization, combining the strengths of gradient-free discrete methods and gradient-based approaches. The discrete optimization in our method is based on low-rank tensor techniques, which, thanks to their low-parameter representation, enable efficient optimization of high-dimensional problems. For the second part, i.e., local search, any effective gradient-based method can be used, whether existing (such as quasi-Newton methods) or any other developed in the future. Our approach addresses the limitations of gradient-based methods, such as getting stuck in local optima; the limitations of discrete methods, which cannot be directly applied to continuous functions; and limitations of gradient-free methods that require large computational budgets. Note that we are not limited to a single type of low-rank tensor decomposition for discrete optimization, but for illustrative purposes, we consider a specific efficient low-rank tensor train decomposition. For 20 challenging 100-dimensional functions, we demonstrate that our method can significantly outperform results obtained with gradient-based methods like Conjugate Gradient, BFGS, SLSQP, and other methods, improving them by orders of magnitude with the same computing budget.

High-dimensional Optimization with Low Rank Tensor Sampling and Local Search

TL;DR

TESALOCS tackles -dimensional, non-convex optimization by fusing a TT-format low-rank discrete surrogate for global exploration with a local optimizer for refinement, updating the TT cores via SGD on a log-likelihood loss. The method achieves scalable exploration with memory that grows linearly in , and it demonstrates improvements over a suite of gradient-based and gradient-free baselines under a fixed budget . Through 20 non-convex, 100-dimensional benchmarks, TESALOCS attains orders-of-magnitude better results, illustrating its effectiveness for challenging high-dimensional problems. The work offers a practical, memory-efficient framework for global-to-local optimization with potential impacts on hyperparameter tuning and materials/design applications.

Abstract

We present a novel method called TESALOCS (TEnsor SAmpling and LOCal Search) for multidimensional optimization, combining the strengths of gradient-free discrete methods and gradient-based approaches. The discrete optimization in our method is based on low-rank tensor techniques, which, thanks to their low-parameter representation, enable efficient optimization of high-dimensional problems. For the second part, i.e., local search, any effective gradient-based method can be used, whether existing (such as quasi-Newton methods) or any other developed in the future. Our approach addresses the limitations of gradient-based methods, such as getting stuck in local optima; the limitations of discrete methods, which cannot be directly applied to continuous functions; and limitations of gradient-free methods that require large computational budgets. Note that we are not limited to a single type of low-rank tensor decomposition for discrete optimization, but for illustrative purposes, we consider a specific efficient low-rank tensor train decomposition. For 20 challenging 100-dimensional functions, we demonstrate that our method can significantly outperform results obtained with gradient-based methods like Conjugate Gradient, BFGS, SLSQP, and other methods, improving them by orders of magnitude with the same computing budget.
Paper Structure (10 sections, 3 equations, 2 figures, 3 tables, 1 algorithm)

This paper contains 10 sections, 3 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. Model.
  4. Sample.
  5. Transform.
  6. Local search.
  7. Update.
  8. Numerical experiments
  9. Related work
  10. Conclusions

Figures (2)

  • Figure 1: Schematic representation of the proposed optimization method TESALOCS.
  • Figure 2: Comparative minimization results for six optimization algorithms (Newton-CG, SLSQP, TNC, BFGS, CG, and L-BFGS-B) applied to the $100$-dimensional Ackley (left), Rosenbrock (middle), and Yang (right) functions. For each method, the graphs display the average minimum value over $10$ runs, depending on the number of queries made to the target function. The conventional baseline approach (random initialization without enhancements) is indicated by dashed lines, while results from our proposed TESALOCS method are shown with solid lines.