Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tailoring Reproducing Kernels for Optimal Control via Policy Iteration

Shengyuan Niu, Ali Bouland, Haoran Wang, Filippos Fotiadis, Andrew Kurdila, Andrea L'Afflitto, Sai Tej Paruchuri, Kyriakos G. Vamvoudakis

TL;DR

The paper develops a framework for tailoring reproducing kernel Hilbert spaces to nonlinear optimal control by embedding policy iteration in native spaces and deriving explicit, stepwise error bounds for both critic and actor updates. By constructing feature mappings that align RKHSs with the problem dynamics, it provides two Galerkin critic schemes (C1 with $Y=W$ and C2 with $Y=L^2(\Omega)$) and analyzes their error contributions, including sampling and approximation terms, with and without Bernstein inequalities. The work extends prior RKHS-based convergence analyses to include the actor update, yielding practical bounds that inform basis selection and conditioning. Numerical experiments on a nonlinear 2D system illustrate that smoother kernels improve final PI accuracy, though they can increase conditioning and computational cost, highlighting trade-offs between C1 and C2 and the potential for adaptive basis strategies. Overall, the approach bridges optimal control and statistical learning by providing provable, geometry-aware error bounds for kernel-based policy iteration in continuous-time settings.

Abstract

This paper presents a novel approach to formulating the actor-critic method for optimal control by casting policy iteration in reproducing kernel Hilbert spaces (RKHSs -- also known as native spaces). By tailoring the reproducing kernel and RKHS to the dynamics of the nonlinear optimal control problem, we leverage recent advancements in characterizing error bounds from statistical and machine learning theory. These approximations define a general strategy to select the bases of the actor-critic networks, and we formally guarantee for the first time that this basis selection procedure leads to closed-form error bounds for the individual steps of policy iteration. These bounds often have a geometric and computable form, making them potentially useful for a priori or a posteriori evaluation of candidate collections of scattered bases. Numerical studies subsequently provide qualitative evidence of the practical performance achieved for the full recursion using the algorithms and theory developed for the single-step error bounds.

Tailoring Reproducing Kernels for Optimal Control via Policy Iteration

TL;DR

The paper develops a framework for tailoring reproducing kernel Hilbert spaces to nonlinear optimal control by embedding policy iteration in native spaces and deriving explicit, stepwise error bounds for both critic and actor updates. By constructing feature mappings that align RKHSs with the problem dynamics, it provides two Galerkin critic schemes (C1 with and C2 with ) and analyzes their error contributions, including sampling and approximation terms, with and without Bernstein inequalities. The work extends prior RKHS-based convergence analyses to include the actor update, yielding practical bounds that inform basis selection and conditioning. Numerical experiments on a nonlinear 2D system illustrate that smoother kernels improve final PI accuracy, though they can increase conditioning and computational cost, highlighting trade-offs between C1 and C2 and the potential for adaptive basis strategies. Overall, the approach bridges optimal control and statistical learning by providing provable, geometry-aware error bounds for kernel-based policy iteration in continuous-time settings.

Abstract

This paper presents a novel approach to formulating the actor-critic method for optimal control by casting policy iteration in reproducing kernel Hilbert spaces (RKHSs -- also known as native spaces). By tailoring the reproducing kernel and RKHS to the dynamics of the nonlinear optimal control problem, we leverage recent advancements in characterizing error bounds from statistical and machine learning theory. These approximations define a general strategy to select the bases of the actor-critic networks, and we formally guarantee for the first time that this basis selection procedure leads to closed-form error bounds for the individual steps of policy iteration. These bounds often have a geometric and computable form, making them potentially useful for a priori or a posteriori evaluation of candidate collections of scattered bases. Numerical studies subsequently provide qualitative evidence of the practical performance achieved for the full recursion using the algorithms and theory developed for the single-step error bounds.
Paper Structure (26 sections, 6 theorems, 55 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 26 sections, 6 theorems, 55 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Notation
  3. Problem Formulation
  4. Characterizing the Range of Policy Iteration Operators in Native Spaces
  5. Feature Mappings, Feature Operators, and Feature Spaces
  6. A Study of the Range of $A\triangleq A(\mu),A^*\triangleq A^*(\mu)$ and $A^*A\triangleq A^*A(\mu)$
  7. A study of the case when A in L(H,W) with W=Ran(A)
  8. A study of the case when A ∈ L(H, L2(Ω)), with R(A) → L2(Ω)
  9. The Range of B
  10. Galerkin Approximations for Policy Iteration
  11. Approximations of the Critic Update Step
  12. Critic Method (C1), when $Y={\mathcal{W}}=\mathcal{R}(A)$
  13. Critic Method (C2), when Y=L2(Omega)
  14. Error Bounds for the Critic
  15. Error Bounds for the Critic Method (C1)
  16. ...and 11 more sections

Key Result

Theorem 1

Let ${\mathbb{H}}$ be a real Hilbert space, $\Phi : \Omega\to {\mathbb{H}}$ be a feature map, $\mathfrak{K}_\Phi$ be the associated kernel defined as in eqn_kernel_feature_map, and $\mathcal{H}_\Phi$ be the real RKHS induced by the kernel $\mathfrak{K}_\Phi$. Then, with The feature operator $F:{\mathbb{H}}\to \mathcal{H}_\Phi$ defined so that $Fh\triangleq (\Phi(\cdot),h)_{\mathbb{H}}$ for any $

Figures (4)

  • Figure 1: Convergence results of PI using the (C1) Method. In these figures the maximum number of iterations $\bar{\ell}$ in Algorithm \ref{['algorigm_PI_2']} is set to $\bar{\ell}=40$
  • Figure 2: Convergence results of PI using Method($C2$). In these figures, the target number of iterations $\bar{\ell}$ in Algorithm \ref{['algorigm_PI_2']} is set to $\bar{\ell}=40$
  • Figure 3: Transient simulation with learned control from Method $(C1)$
  • Figure 4: Transient simulation with learned control from Method $(C2)$

Theorems & Definitions (9)

  • Theorem 1: xu2007refinable, carmeli2005reproducing
  • Theorem 2
  • proof
  • Theorem 3
  • proof
  • Theorem 4
  • proof
  • Theorem 5: zhou2008derivative
  • Theorem 6: kirsch2011introductionkress1989linear