Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chaos-based reinforcement learning with TD3

Toshitaka Matsuki, Yusuke Sakemi, Kazuyuki Aihara

TL;DR

The paper tackles enabling chaos-driven exploration in reinforcement learning by pairing a TD3-based off-policy algorithm with an echo-state reservoir to drive internal chaotic dynamics. It demonstrates that chaotic reservoir dynamics can drive exploration without external noise, support autonomous suppression of exploration as learning improves, and enable re-learning when environmental rules change, all within a simple goal-reaching setting. The results reveal an optimal range of chaoticity (spectral radius) for balancing exploration and exploitation, with excessive chaos hindering learning in larger state spaces and more difficult tasks. The findings highlight both the promise and current limitations of CBRL, suggesting future work on richer reservoir architectures, chaos quantification, and hybrid exploration strategies to extend to high-dimensional tasks.

Abstract

Chaos-based reinforcement learning (CBRL) is a method in which the agent's internal chaotic dynamics drives exploration. However, the learning algorithms in CBRL have not been thoroughly developed in previous studies, nor have they incorporated recent advances in reinforcement learning. This study introduced Twin Delayed Deep Deterministic Policy Gradients (TD3), which is one of the state-of-the-art deep reinforcement learning algorithms that can treat deterministic and continuous action spaces, to CBRL. The validation results provide several insights. First, TD3 works as a learning algorithm for CBRL in a simple goal-reaching task. Second, CBRL agents with TD3 can autonomously suppress their exploratory behavior as learning progresses and resume exploration when the environment changes. Finally, examining the effect of the agent's chaoticity on learning shows that there exists a suitable range of chaos strength in the agent's model to flexibly switch between exploration and exploitation and adapt to environmental changes.

Chaos-based reinforcement learning with TD3

TL;DR

The paper tackles enabling chaos-driven exploration in reinforcement learning by pairing a TD3-based off-policy algorithm with an echo-state reservoir to drive internal chaotic dynamics. It demonstrates that chaotic reservoir dynamics can drive exploration without external noise, support autonomous suppression of exploration as learning improves, and enable re-learning when environmental rules change, all within a simple goal-reaching setting. The results reveal an optimal range of chaoticity (spectral radius) for balancing exploration and exploitation, with excessive chaos hindering learning in larger state spaces and more difficult tasks. The findings highlight both the promise and current limitations of CBRL, suggesting future work on richer reservoir architectures, chaos quantification, and hybrid exploration strategies to extend to high-dimensional tasks.

Abstract

Chaos-based reinforcement learning (CBRL) is a method in which the agent's internal chaotic dynamics drives exploration. However, the learning algorithms in CBRL have not been thoroughly developed in previous studies, nor have they incorporated recent advances in reinforcement learning. This study introduced Twin Delayed Deep Deterministic Policy Gradients (TD3), which is one of the state-of-the-art deep reinforcement learning algorithms that can treat deterministic and continuous action spaces, to CBRL. The validation results provide several insights. First, TD3 works as a learning algorithm for CBRL in a simple goal-reaching task. Second, CBRL agents with TD3 can autonomously suppress their exploratory behavior as learning progresses and resume exploration when the environment changes. Finally, examining the effect of the agent's chaoticity on learning shows that there exists a suitable range of chaos strength in the agent's model to flexibly switch between exploration and exploitation and adapt to environmental changes.
Paper Structure (26 sections, 13 equations, 75 figures)

This paper contains 26 sections, 13 equations, 75 figures.

Table of Contents

  1. Introduction
  2. Chaos-based reinforcement learning
  3. Exploration driven by internal chaotic dynamics
  4. Expectation for CBRL
  5. Issue of learning algorithm for CBRL
  6. Method
  7. TD3
  8. Reservoir network
  9. TD3-CBRL
  10. Goal task
  11. Experiment
  12. Conditions
  13. Learning result
  14. Effects by presence or absence of exploration component
  15. Goal change task.
  16. ...and 11 more sections

Figures (75)

  • Figure 1: Regular reinforcement learning.
  • Figure 2: Chaos-based reinforcement learning.
  • Figure 4: Echo state network (ESN). An ESN has a reservoir that is a special recurrent layer whose recurrent and input weights are randomly and sparsely connected and fixed. Only the weights of the output layer (indicated by the red arrows) are trained.
  • Figure 5: TD3-CBRL
  • Figure 6: TD3 without external exploration noise
  • ...and 70 more figures