Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Technical Report on Reinforcement Learning Control on the Lucas-Nülle Inverted Pendulum

Maximilian Schenke, Shalbus Bukarov

TL;DR

This work demonstrates reinforcement learning-based control for swing-up and stabilization of an inverted pendulum on educational hardware, using a Deep Deterministic Policy Gradient (DDPG) framework with separate actor and critic networks. The return $g$ is maximized with $g_k = r_{k+1} + \gamma r_{k+2} + \dots$ and the optimal action given by $a^*_k = \arg\max_a q(\mathbf{o}_k,a)$, while observing a Markov state through a designed feature vector and velocity estimates obtained via a PLL-based estimator. Experiments show convergence over 30 minutes of training and successful swing-up with safeguarding constraints, though exact $x_{ref}$ tracking remains challenging; safeguarding is shown to be essential during training, and future work points to faster convergence and reward redesign to reduce reliance on safety constraints. The approach provides an educative, model-free alternative to classical controllers, with practical implications for hands-on RL in control education.

Abstract

The discipline of automatic control is making increased use of concepts that originate from the domain of machine learning. Herein, reinforcement learning (RL) takes an elevated role, as it is inherently designed for sequential decision making, and can be applied to optimal control problems without the need for a plant system model. To advance education of control engineers and operators in this field, this contribution targets an RL framework that can be applied to educational hardware provided by the Lucas-Nülle company. Specifically, the goal of inverted pendulum control is pursued by means of RL, including both, swing-up and stabilization within a single holistic design approach. Herein, the actual learning is enabled by separating corresponding computations from the real-time control computer and outsourcing them to a different hardware. This distributed architecture, however, necessitates communication of the involved components, which is realized via CAN bus. The experimental proof of concept is presented with an applied safeguarding algorithm that prevents the plant from being operated harmfully during the trial-and-error training phase.

Technical Report on Reinforcement Learning Control on the Lucas-Nülle Inverted Pendulum

TL;DR

This work demonstrates reinforcement learning-based control for swing-up and stabilization of an inverted pendulum on educational hardware, using a Deep Deterministic Policy Gradient (DDPG) framework with separate actor and critic networks. The return is maximized with and the optimal action given by , while observing a Markov state through a designed feature vector and velocity estimates obtained via a PLL-based estimator. Experiments show convergence over 30 minutes of training and successful swing-up with safeguarding constraints, though exact tracking remains challenging; safeguarding is shown to be essential during training, and future work points to faster convergence and reward redesign to reduce reliance on safety constraints. The approach provides an educative, model-free alternative to classical controllers, with practical implications for hands-on RL in control education.

Abstract

The discipline of automatic control is making increased use of concepts that originate from the domain of machine learning. Herein, reinforcement learning (RL) takes an elevated role, as it is inherently designed for sequential decision making, and can be applied to optimal control problems without the need for a plant system model. To advance education of control engineers and operators in this field, this contribution targets an RL framework that can be applied to educational hardware provided by the Lucas-Nülle company. Specifically, the goal of inverted pendulum control is pursued by means of RL, including both, swing-up and stabilization within a single holistic design approach. Herein, the actual learning is enabled by separating corresponding computations from the real-time control computer and outsourcing them to a different hardware. This distributed architecture, however, necessitates communication of the involved components, which is realized via CAN bus. The experimental proof of concept is presented with an applied safeguarding algorithm that prevents the plant from being operated harmfully during the trial-and-error training phase.

Paper Structure

This paper contains 16 sections, 31 equations, 8 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Setting
  3. Reinforcement Learning Control
  4. Deep Deterministic Policy Gradient (DDPG)
  5. Estimating $q$
  6. Estimating $a^*$
  7. Exploration Noise
  8. Observation Design / Feature Engineering
  9. Reward Design
  10. Safeguarding
  11. Implementation
  12. Experimental Results
  13. Training Phase
  14. Application Phase
  15. Conclusion and Outlook
  16. ...and 1 more sections

Figures (8)

  • Figure 1: Schematic depiction of the closed-loop control system
  • Figure 2: Conceptual depiction of the inverted pendulum
  • Figure 3: Depiction of the reward gradients with respect to angle $\theta$ and angular velocity $\hat{\omega}$, the green shaded area corresponds to safeguard activation, the yellow shaded area corresponds to optimal operation
  • Figure 4: Schematic of the employed online RL setup
  • Figure 5: Left: average convergence behavior of five different RL controllers trained with the pendulum weight in the innermost position, Right: average convergence behavior of five different RL controllers trained with the pendulum weight in the outermost position; the blue shaded area denotes one standard deviation
  • ...and 3 more figures