Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dyna-Style Reinforcement Learning Modeling and Control of Non-linear Dynamics

Karim Abdelsalam, Zeyad Gamal, Ayman El-Badawy

TL;DR

This work tackles data-efficient control of nonlinear dynamics by fusing sparse identification of nonlinear dynamics (SINDy) with TD3 in a Dyna-style framework. Real interactions are augmented with synthetic rollouts generated by a learned SINDy model, enabling rapid policy improvement while reducing hardware wear and data requirements. The bi-rotor case study shows that SINDy-TD3 achieves superior tracking accuracy and robustness with significantly fewer real interactions than direct TD3, across step, sine, and square trajectories. The approach combines interpretable, physics-informed modeling with powerful deep RL, offering a practical pathway to safe and efficient control of complex nonlinear systems.

Abstract

Controlling systems with complex, nonlinear dynamics poses a significant challenge, particularly in achieving efficient and robust control. In this paper, we propose a Dyna-Style Reinforcement Learning control framework that integrates Sparse Identification of Nonlinear Dynamics (SINDy) with Twin Delayed Deep Deterministic Policy Gradient (TD3) reinforcement learning. SINDy is used to identify a data-driven model of the system, capturing its key dynamics without requiring an explicit physical model. This identified model is used to generate synthetic rollouts that are periodically injected into the reinforcement learning replay buffer during training on the real environment, enabling efficient policy learning with limited data available. By leveraging this hybrid approach, we mitigate the sample inefficiency of traditional model-free reinforcement learning methods while ensuring accurate control of nonlinear systems. To demonstrate the effectiveness of this framework, we apply it to a bi-rotor system as a case study, evaluating its performance in stabilization and trajectory tracking. The results show that our SINDy-TD3 approach achieves superior accuracy and robustness compared to direct reinforcement learning techniques, highlighting the potential of combining data-driven modeling with reinforcement learning for complex dynamical systems.

Dyna-Style Reinforcement Learning Modeling and Control of Non-linear Dynamics

TL;DR

This work tackles data-efficient control of nonlinear dynamics by fusing sparse identification of nonlinear dynamics (SINDy) with TD3 in a Dyna-style framework. Real interactions are augmented with synthetic rollouts generated by a learned SINDy model, enabling rapid policy improvement while reducing hardware wear and data requirements. The bi-rotor case study shows that SINDy-TD3 achieves superior tracking accuracy and robustness with significantly fewer real interactions than direct TD3, across step, sine, and square trajectories. The approach combines interpretable, physics-informed modeling with powerful deep RL, offering a practical pathway to safe and efficient control of complex nonlinear systems.

Abstract

Controlling systems with complex, nonlinear dynamics poses a significant challenge, particularly in achieving efficient and robust control. In this paper, we propose a Dyna-Style Reinforcement Learning control framework that integrates Sparse Identification of Nonlinear Dynamics (SINDy) with Twin Delayed Deep Deterministic Policy Gradient (TD3) reinforcement learning. SINDy is used to identify a data-driven model of the system, capturing its key dynamics without requiring an explicit physical model. This identified model is used to generate synthetic rollouts that are periodically injected into the reinforcement learning replay buffer during training on the real environment, enabling efficient policy learning with limited data available. By leveraging this hybrid approach, we mitigate the sample inefficiency of traditional model-free reinforcement learning methods while ensuring accurate control of nonlinear systems. To demonstrate the effectiveness of this framework, we apply it to a bi-rotor system as a case study, evaluating its performance in stabilization and trajectory tracking. The results show that our SINDy-TD3 approach achieves superior accuracy and robustness compared to direct reinforcement learning techniques, highlighting the potential of combining data-driven modeling with reinforcement learning for complex dynamical systems.
Paper Structure (19 sections, 12 equations, 15 figures, 2 tables, 1 algorithm)

This paper contains 19 sections, 12 equations, 15 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Dyna-Style SINDY-TD3 Framework
  3. Training Data Collection
  4. Model Learning
  5. Reinforcement Learning Control (TD3)
  6. Case Study: Bi-Rotor
  7. Data-Driven Model
  8. Training Data Generation
  9. Sparsity Coefficient and Library Selection
  10. TD3 Trajectory Tracking Agent
  11. Results and Discussion
  12. Training SINDy Model
  13. Reinforcement Learning Control
  14. Scenario A: Step Trajectory
  15. Scenario B: Sine Wave Trajectory
  16. ...and 4 more sections

Figures (15)

  • Figure 1: Dyna-Style SINDy-TD3 Reinforcement Learning Architecture
  • Figure 2: SINDy-TD3 control framework
  • Figure 3: Bi-rotor lab setup Ragi
  • Figure 4: Bi-rotor schematic diagram Inteco2006
  • Figure 5: Chirp signal inputs to the Bi-rotor
  • ...and 10 more figures