Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings

Hongpeng Cao, Yanbing Mao, Lui Sha, Marco Caccamo

TL;DR

Phy-DRL tackles safety in learning-based autonomy by integrating physics with deep RL through three invariant embeddings: a Residual Action Policy that linearizes part of the control and stabilizes learning, a Safety-Embedded Reward that enforces a Lyapunov-like safety criterion via a scalar envelope ${\Omega}$ and LMIs to maximize safety, and Physics-Knowledge-Enhanced DNNs (PhyN) that enforce partial physics knowledge on the critic and actor via Taylor-series augmentation and knowledge-based editing. The authors prove a safety and stability guarantee under certain conditions and demonstrate faster, more robust training on cart-pole and quadruped tasks, outperforming purely data-driven DRL and pure model-based controllers in safety and efficiency. The framework offers a practical path toward safe, sample-efficient RL for safety-critical robotics, with reproducible results and publicly available code. Overall, Phy-DRL integrates physics priors with invariant architectural designs to deliver provable safety while maintaining learning flexibility.

Abstract

This paper proposes the Phy-DRL: a physics-regulated deep reinforcement learning (DRL) framework for safety-critical autonomous systems. The Phy-DRL has three distinguished invariant-embedding designs: i) residual action policy (i.e., integrating data-driven-DRL action policy and physics-model-based action policy), ii) automatically constructed safety-embedded reward, and iii) physics-model-guided neural network (NN) editing, including link editing and activation editing. Theoretically, the Phy-DRL exhibits 1) a mathematically provable safety guarantee and 2) strict compliance of critic and actor networks with physics knowledge about the action-value function and action policy. Finally, we evaluate the Phy-DRL on a cart-pole system and a quadruped robot. The experiments validate our theoretical results and demonstrate that Phy-DRL features guaranteed safety compared to purely data-driven DRL and solely model-based design while offering remarkably fewer learning parameters and fast training towards safety guarantee.

Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings

TL;DR

Phy-DRL tackles safety in learning-based autonomy by integrating physics with deep RL through three invariant embeddings: a Residual Action Policy that linearizes part of the control and stabilizes learning, a Safety-Embedded Reward that enforces a Lyapunov-like safety criterion via a scalar envelope and LMIs to maximize safety, and Physics-Knowledge-Enhanced DNNs (PhyN) that enforce partial physics knowledge on the critic and actor via Taylor-series augmentation and knowledge-based editing. The authors prove a safety and stability guarantee under certain conditions and demonstrate faster, more robust training on cart-pole and quadruped tasks, outperforming purely data-driven DRL and pure model-based controllers in safety and efficiency. The framework offers a practical path toward safe, sample-efficient RL for safety-critical robotics, with reproducible results and publicly available code. Overall, Phy-DRL integrates physics priors with invariant architectural designs to deliver provable safety while maintaining learning flexibility.

Abstract

This paper proposes the Phy-DRL: a physics-regulated deep reinforcement learning (DRL) framework for safety-critical autonomous systems. The Phy-DRL has three distinguished invariant-embedding designs: i) residual action policy (i.e., integrating data-driven-DRL action policy and physics-model-based action policy), ii) automatically constructed safety-embedded reward, and iii) physics-model-guided neural network (NN) editing, including link editing and activation editing. Theoretically, the Phy-DRL exhibits 1) a mathematically provable safety guarantee and 2) strict compliance of critic and actor networks with physics knowledge about the action-value function and action policy. Finally, we evaluate the Phy-DRL on a cart-pole system and a quadruped robot. The experiments validate our theoretical results and demonstrate that Phy-DRL features guaranteed safety compared to purely data-driven DRL and solely model-based design while offering remarkably fewer learning parameters and fast training towards safety guarantee.
Paper Structure (54 sections, 6 theorems, 101 equations, 18 figures, 3 tables, 2 algorithms)

This paper contains 54 sections, 6 theorems, 101 equations, 18 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Motivations
  3. Contributions
  4. Related Work and Open Problems
  5. Summary: Answers to \ref{['defa1a']}--\ref{['defa3']}
  6. Preliminaries
  7. Dynamics Model of Real Plant
  8. Safety Definition
  9. Design Overview: Invariant Embeddings
  10. Invariant Embedding 1: Residual Action Policy
  11. Invariant Embedding 2: Safety-Embedded Reward
  12. Invariant Embedding 3: Physics-Knowledge-Enhanced DNN
  13. Experiments
  14. Cart-Pole System
  15. Quadruped Robot
  16. ...and 39 more sections

Key Result

Lemma 5.1

Consider the sets defined in aset2 and set3. We have ${\Omega} \subseteq \mathbb{X}$, if where $\overline{\mathbf{D}} = \frac{{\mathbf{D}}}{\overline{\Lambda}}$, $\underline{\mathbf{D}} = \frac{{\mathbf{D}}}{\underline{\Lambda}}$, and $\mathbf{d}$, $\overline{\Lambda}$ and $\underline{\Lambda}$ are defined below for $i,j \in \{1, 2, \ldots, h\}$:

Figures (18)

  • Figure 1: Phy-DRL Framework, applied to a quadruped robot.
  • Figure 2: Physics-Knowledge-Enhanced DNN architecture.
  • Figure 3: Blue: area of IE samples defined in \ref{['iess']}. Green: area of EE samples defined in \ref{['eess']}. Rectangular area: safety set. Ellipse area: safety envelope.
  • Figure 4: Phase plots of models running different environments, given different velocity commands.
  • Figure 5: Illustration of the proof path of Theorem \ref{['thm1']}.
  • ...and 13 more figures

Theorems & Definitions (18)

  • Definition 2.1
  • Remark 2.2: Role of $\Omega$
  • Lemma 5.1
  • Remark 5.2: Sub-rewards
  • Theorem 5.3: Mathematically Provable Safety Guarantee
  • Remark 5.4: Solving Optimal $\mathbf{R}$ and $\mathbf{Q}$
  • Remark 5.5: $\mathbf{F}$ is given
  • Remark 5.6: Provable Stability Guarantee
  • Remark 5.7: Fast Training
  • Remark 5.8: Obtaining ($\mathbf{A}$, $\mathbf{B}$)
  • ...and 8 more