Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kernel-Based Optimal Control: An Infinitesimal Generator Approach

Petar Bevanda, Nicolas Hoischen, Tobias Wittmann, Jan Brüdigam, Sandra Hirche, Boris Houska

TL;DR

The paper addresses data-driven optimal control for nonlinear stochastic difusions by learning the infinitesimal generator of controlled diffusions in an RKHS. It develops a Kernel Hamilton-Jacobi-Bellman framework ($KHJB$) that leverages convex reformulations of the ergodic control problem via a PDE-constrained dual, enabling global-ish solutions without explicit parametric models. The methodology hinges on Hilbert-Schmidt regression in infinite dimensions to nonparametrically estimate the adjoints of the generator, producing an explicit finite-dimensional backward value function and feedback policy through a data-driven KHJB recursion. Numerical results on synthetic oscillators and robotic systems demonstrate superior robustness and performance against modern data-driven and classical nonlinear programming methods, highlighting practical impact for scalable, model-free control under uncertainty.

Abstract

This paper presents a novel operator-theoretic approach for optimal control of nonlinear stochastic systems within reproducing kernel Hilbert spaces. Our learning framework leverages data samples of system dynamics and stage cost functions, with only control penalties and constraints provided. The proposed method directly learns the infinitesimal generator of a controlled stochastic diffusion in an infinite-dimensional hypothesis space. We demonstrate that our approach seamlessly integrates with modern convex operator-theoretic Hamilton-Jacobi-Bellman recursions, enabling a data-driven solution to the optimal control problems. Furthermore, our learning framework includes nonparametric estimators for uncontrolled infinitesimal generators as a special case. Numerical experiments, ranging from synthetic differential equations to simulated robotic systems, showcase the advantages of our approach compared to both modern data-driven and classical nonlinear programming methods for optimal control.

Kernel-Based Optimal Control: An Infinitesimal Generator Approach

TL;DR

The paper addresses data-driven optimal control for nonlinear stochastic difusions by learning the infinitesimal generator of controlled diffusions in an RKHS. It develops a Kernel Hamilton-Jacobi-Bellman framework () that leverages convex reformulations of the ergodic control problem via a PDE-constrained dual, enabling global-ish solutions without explicit parametric models. The methodology hinges on Hilbert-Schmidt regression in infinite dimensions to nonparametrically estimate the adjoints of the generator, producing an explicit finite-dimensional backward value function and feedback policy through a data-driven KHJB recursion. Numerical results on synthetic oscillators and robotic systems demonstrate superior robustness and performance against modern data-driven and classical nonlinear programming methods, highlighting practical impact for scalable, model-free control under uncertainty.

Abstract

This paper presents a novel operator-theoretic approach for optimal control of nonlinear stochastic systems within reproducing kernel Hilbert spaces. Our learning framework leverages data samples of system dynamics and stage cost functions, with only control penalties and constraints provided. The proposed method directly learns the infinitesimal generator of a controlled stochastic diffusion in an infinite-dimensional hypothesis space. We demonstrate that our approach seamlessly integrates with modern convex operator-theoretic Hamilton-Jacobi-Bellman recursions, enabling a data-driven solution to the optimal control problems. Furthermore, our learning framework includes nonparametric estimators for uncontrolled infinitesimal generators as a special case. Numerical experiments, ranging from synthetic differential equations to simulated robotic systems, showcase the advantages of our approach compared to both modern data-driven and classical nonlinear programming methods for optimal control.

Paper Structure

This paper contains 14 sections, 2 theorems, 21 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contribution.
  3. Problem Statement
  4. Operator-Theoretic Dynamic Programming and HJB Recursions
  5. Convex Reformulation
  6. Hamilton-Jacobi-Bellman equations
  7. Generator Regression for Control-Affine Diffusions and HJB Approximation
  8. Reproducing Kernel Hilbert Spaces
  9. Hilbert-Schmidt regression in infinite dimensions
  10. Numerical Experiments
  11. Unstable Oscillator
  12. Torque Limited Inverted Pendulum
  13. Cartpole: Inverted Pendulum on a Cart
  14. Conclusion

Key Result

lemma 1

Let $k$ be a Mercer kernel such that $k \in C^4(\mathbb{X} \times \mathbb{X})$ with corresponding RKHS $\mathcal{H}$ and the system dynamics be described by eq::ctrlSDE under inputs $\bm{\pi}~\in~\bm{0} \cup \{\bm{e}_j\}_{j \in [n_u]}$. Then, the entries of the target kernel matrices are computed vi

Figures (3)

  • Figure 1: Comparison of RMSE to the known optimal policy $\boldsymbol{\pi}_{\infty}^\star(\bm{x})$ between KHJB ( bevanda2024data) and our IG-KHJB approach for the Van der Pol Oscillator.
  • Figure 2: Contour plots of the value and controller functions learned from $2.5\cdot10^3$ samples using our IG-KHJB approach (upright at $(\dot{\theta},\theta){=}(0,0)$).
  • Figure 3: Accumulated stage costs using our learned policy $\widehat{\bm{\pi}}^\star({\bm{x}})$ and $\texttt{Altro-NMPC}$.

Theorems & Definitions (4)

  • lemma 1
  • proof
  • proposition 1
  • proof