Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

State-dependent temperature control in Langevin diffusions using numerical exploratory Hamiltonian-Jacobi-Bellman equations

Taorui Wang, Xun Li, Gu Wang, Zhongqiang Zhang

Abstract

Choosing how much noise to add in Langevin dynamics is essential for making these algorithms effective in challenging optimization problems. One promising approach is to determine this noise by solving Hamilton-Jacobi-Bellman (HJB) equations and their exploratory variants. Though these ideas have been demonstrated to work well in one dimension, extension to high-dimensional minimization has been limited by two unresolved numerical challenges: setting reliable control bounds and stably computing the second-order information (Hessians) required by the equations. These issues and the broader impact of HJB parameters have not been systematically examined. This work provides the first such investigation. We introduce principled control bounds and develop a physics-informed neural network framework that embeds the structure of exploratory HJB equations directly into training, stabilizing computation, and enabling accurate estimation of state-dependent noise in high-dimensional problems. Numerical experiments demonstrate that the resulting method remains robust and effective well beyond low-dimensional test cases.

State-dependent temperature control in Langevin diffusions using numerical exploratory Hamiltonian-Jacobi-Bellman equations

Abstract

Choosing how much noise to add in Langevin dynamics is essential for making these algorithms effective in challenging optimization problems. One promising approach is to determine this noise by solving Hamilton-Jacobi-Bellman (HJB) equations and their exploratory variants. Though these ideas have been demonstrated to work well in one dimension, extension to high-dimensional minimization has been limited by two unresolved numerical challenges: setting reliable control bounds and stably computing the second-order information (Hessians) required by the equations. These issues and the broader impact of HJB parameters have not been systematically examined. This work provides the first such investigation. We introduce principled control bounds and develop a physics-informed neural network framework that embeds the structure of exploratory HJB equations directly into training, stabilizing computation, and enabling accurate estimation of state-dependent noise in high-dimensional problems. Numerical experiments demonstrate that the resulting method remains robust and effective well beyond low-dimensional test cases.
Paper Structure (13 sections, 4 theorems, 38 equations, 8 figures, 5 tables, 3 algorithms)

This paper contains 13 sections, 4 theorems, 38 equations, 8 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Literature review
  3. Problem Setup
  4. Methodology
  5. PINNs for the exploratory HJB equation
  6. Stable evaluation of $H_\lambda(\nabla^2 v_{\lambda,\phi})$ and $h_{\lambda,\phi}$
  7. Choice of the control set $\mathcal{U}$ and the discount rate $\rho$
  8. Langevin dynamics with approximated temperature
  9. Numerical Examples
  10. Conclusion and discussion
  11. Monotone finite differences and Howard policy iteration for the classical HJB equation
  12. Additional tests for Easom and Hartmann-6
  13. Proofs

Key Result

Theorem 4.1

\newlabelthm:convergence-lambda-pertub0 Let $\Omega \subset\mathbb{R}^d$ be an open bounded domain with a smooth boundary. Assume that $f\in C^2(\Omega)$ and $\widetilde{R}_\lambda(\bm{x};\phi)\in L^\infty (\Omega)$ and is uniformly bounded in $\lambda$. Let $v$ and $v_{\lambda,\phi}$ satisfy solu

Figures (8)

  • Figure 1: One-dimensional test: $\Delta v$ and $\Delta v_{\lambda,\phi}$ for $\lambda =\frac{0.32}{2^{p}},\,p=0,1,\ldots, 4$.
  • Figure 1: Additional tests of the errors $\mathcal{E}_k$ of numerical global minimizers with various $\lambda$'s. (a) Easom with $(u_{\min},u_{\max},\rho,\tau)=(10^{-8},16\kappa,0.02,0)$ and step size $\eta=0.128$. (b) Hartmann-6 with $(u_{\min},u_{\max},\rho,\tau)=(10^{-8},\kappa,1.6,0.02)$ and step size $\eta=0.016$. In both cases, $\lambda=0.01,0.02,0.04,0.08,0.16$.
  • Figure 2: Example \ref{['exm:double-well']} (1D double-well). (a) Objective $f$. (b) $v_{\lambda,\phi}"$ for multiple $\lambda$ compared with the reference $v"$ (Appendix \ref{['app:fd-howard']}). (c) $h_{\lambda,\phi}$ compared with $\sqrt{2\bar{u}^*}$ from \ref{['eq:class-opt-control']}.
  • Figure 3: 1D double-well: $\hat{f}_k$ (100-trajectory average) for multiple $\lambda$. Left: $\tau=0$. Right: $\tau=\tfrac{1}{2}\sqrt{2u_{\max}}$.
  • Figure 4: Example \ref{['exm:2d-gauss-mix']}: 2D Gaussian mixture on $\Omega=[-1,5]^2$ (contour and surface). The landscape contains multiple wells; the deepest well is near $(3,2)$. Left: contour plot of $f$; right: surface plot.
  • ...and 3 more figures

Theorems & Definitions (12)

  • Theorem 4.1
  • Remark 4.2
  • Example 5.1: Testing PINNs for eHJB equation
  • Example 5.2: 1D double well, GaoXuZhou2022, evaluating noise coefficient
  • Example 5.3: 2D Gaussian mixture, dong2021replicaexchangenonconvexoptimization
  • Example 5.4: Easom function, minima in a plateau
  • Example 5.5: Hartmann 6D function
  • Lemma C.1
  • Proof 1
  • Theorem C.2
  • ...and 2 more