Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence of machine learning methods for feedback control laws: averaged feedback learning scheme and data driven methods

Karl Kunisch, Donato Vásquez-Varas

TL;DR

This work analyzes how two machine learning approaches—AFLS and data-driven regression—for constructing optimal feedback laws converge to the true value function in control problems. AFLS requires only Hölder regularity of the value function, while the data-driven methods demand at least $C^2$ smoothness; the authors connect the approaches via their optimality conditions and corroborate theory with a parameterized family where value-function regularity is tunable. They establish rigorous convergence results for finite-horizon approximations as the horizon grows and show how the choice of regularity impacts performance, with AFLS performing better when the value function is non-differentiable but semi-concave. The results have practical implications for scalable feedback design in high-dimensional systems, guiding which learning paradigm to employ given the regularity of the underlying value function.

Abstract

This work addresses the synthesis of optimal feedback control laws via machine learning. In particular, the Averaged Feedback Learning Scheme (AFLS) and a data driven method are considered. Hypotheses for each method ensuring the convergence of the evaluation of the objective function of the underlying control problem at the obtained feedback-laws towards the optimal value function are provided. These hypotheses are connected to the regularity of the value function and the stability of the dynamics. In the case of AFLS these hypotheses only require Hölder continuity of the value function, whereas for the data driven method the value function must be at least $C^2$. It is demonstrated that these methods are connected via their optimality conditions. Additionally, numerical experiments are provided by applying both methods to a family control problems, parameterized by a positive real number which controls the regularity of the value function. For small parameters the value function is smooth and in contrast for large parameters it is non-differentiable, but semi-concave. The results of the experiments indicate that both methods have a similar performance for the case that the value function is smooth. On the other hand, if the value function is not differentiable, AFLS has a better performance which is consistent with the obtained convergence results.

Convergence of machine learning methods for feedback control laws: averaged feedback learning scheme and data driven methods

TL;DR

This work analyzes how two machine learning approaches—AFLS and data-driven regression—for constructing optimal feedback laws converge to the true value function in control problems. AFLS requires only Hölder regularity of the value function, while the data-driven methods demand at least smoothness; the authors connect the approaches via their optimality conditions and corroborate theory with a parameterized family where value-function regularity is tunable. They establish rigorous convergence results for finite-horizon approximations as the horizon grows and show how the choice of regularity impacts performance, with AFLS performing better when the value function is non-differentiable but semi-concave. The results have practical implications for scalable feedback design in high-dimensional systems, guiding which learning paradigm to employ given the regularity of the underlying value function.

Abstract

This work addresses the synthesis of optimal feedback control laws via machine learning. In particular, the Averaged Feedback Learning Scheme (AFLS) and a data driven method are considered. Hypotheses for each method ensuring the convergence of the evaluation of the objective function of the underlying control problem at the obtained feedback-laws towards the optimal value function are provided. These hypotheses are connected to the regularity of the value function and the stability of the dynamics. In the case of AFLS these hypotheses only require Hölder continuity of the value function, whereas for the data driven method the value function must be at least . It is demonstrated that these methods are connected via their optimality conditions. Additionally, numerical experiments are provided by applying both methods to a family control problems, parameterized by a positive real number which controls the regularity of the value function. For small parameters the value function is smooth and in contrast for large parameters it is non-differentiable, but semi-concave. The results of the experiments indicate that both methods have a similar performance for the case that the value function is smooth. On the other hand, if the value function is not differentiable, AFLS has a better performance which is consistent with the obtained convergence results.
Paper Structure (4 sections, 4 theorems, 43 equations)

This paper contains 4 sections, 4 theorems, 43 equations.

Table of Contents

  1. Introduction
  2. Control problem
  3. Methods for the construction of approximating optimal feedback laws
  4. Convergence analysis

Key Result

Theorem 4.1

\newlabeltheo:conv:LearningProblemOpt0 Let Hypotheses hypo:approx:vf and hypo:approx:c2 hold for sequences $(\Theta_n,\left\lVert\cdot\right\rVert_{n})$ , $\mathcal{P}_n$, $v_n$ and $T_n$. Then there exist a sequence $\alpha_{n}\!>\!0$ and a sub-sequence $((\Theta_{k_n},\left\lVert\cdot\right\rVer

Theorems & Definitions (14)

  • Definition 3.1
  • Definition 3.2
  • Remark 3.1
  • Definition 3.5
  • Remark 3.2
  • Remark 3.3
  • Remark 3.4
  • Theorem 4.1
  • Remark 4.1
  • Theorem 4.2
  • ...and 4 more