Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model-Based Data-Efficient and Robust Reinforcement Learning

Ludvig Svedlund, Constantin Cronrath, Jonas Fredriksson, Bengt Lennartson

TL;DR

The paper tackles data efficiency and robustness in reinforcement learning for constrained, energy-aware control. It advocates a modularized, model-based RL framework that separates dynamics estimation from control optimization and leverages temporal optimization to generate energy-efficient reference trajectories. Compared with model-free methods and standard DRL baselines, the approach demonstrates improved energy savings and significantly reduced sample requirements in two electric-vehicle case studies, while highlighting robustness advantages against unmodeled dynamics. The work provides a practical blueprint for deploying data-efficient, constraint-aware RL in mechatronic systems and EV energy optimization, with avenues for extending to nonlinear IO models and broader TO formulations.

Abstract

A data-efficient learning-based control design method is proposed in this paper. It is based on learning a system dynamics model that is then leveraged in a two-level procedure. On the higher level, a simple but powerful optimization procedure is performed such that, for example, energy consumption in a vehicle can be reduced when hard state and action constraints are also introduced. Load disturbances and model errors are compensated for by a feedback controller on the lower level. In that regard, we briefly examine the robustness of both model-free and model-based learning approaches, and it is shown that the model-free approach greatly suffers from the inclusion of unmodeled dynamics. In evaluating the proposed method, it is assumed that a path is given, while the velocity and acceleration can be modified such that energy is saved, while still keeping speed limits and completion time. Compared with two well-known actor-critic reinforcement learning strategies, the suggested learning-based approach saves more energy and reduces the number of evaluated time steps by a factor of 100 or more.

Model-Based Data-Efficient and Robust Reinforcement Learning

TL;DR

The paper tackles data efficiency and robustness in reinforcement learning for constrained, energy-aware control. It advocates a modularized, model-based RL framework that separates dynamics estimation from control optimization and leverages temporal optimization to generate energy-efficient reference trajectories. Compared with model-free methods and standard DRL baselines, the approach demonstrates improved energy savings and significantly reduced sample requirements in two electric-vehicle case studies, while highlighting robustness advantages against unmodeled dynamics. The work provides a practical blueprint for deploying data-efficient, constraint-aware RL in mechatronic systems and EV energy optimization, with avenues for extending to nonlinear IO models and broader TO formulations.

Abstract

A data-efficient learning-based control design method is proposed in this paper. It is based on learning a system dynamics model that is then leveraged in a two-level procedure. On the higher level, a simple but powerful optimization procedure is performed such that, for example, energy consumption in a vehicle can be reduced when hard state and action constraints are also introduced. Load disturbances and model errors are compensated for by a feedback controller on the lower level. In that regard, we briefly examine the robustness of both model-free and model-based learning approaches, and it is shown that the model-free approach greatly suffers from the inclusion of unmodeled dynamics. In evaluating the proposed method, it is assumed that a path is given, while the velocity and acceleration can be modified such that energy is saved, while still keeping speed limits and completion time. Compared with two well-known actor-critic reinforcement learning strategies, the suggested learning-based approach saves more energy and reduces the number of evaluated time steps by a factor of 100 or more.
Paper Structure (45 sections, 42 equations, 4 figures, 5 tables)

This paper contains 45 sections, 42 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Model-free Reinforcement Learning
  3. Linear Quadratic Reinforcement Learning
  4. Quadratic Q-function
  5. Policy iteration
  6. Policy improvement
  7. Model-free strategy
  8. Deep Reinforcement Learning
  9. TD3 and SAC
  10. Model-based Reinforcement Learning
  11. Linear model estimation
  12. ARX models
  13. Linear state-space model
  14. State feedback control design
  15. Nonlinear model estimation
  16. ...and 30 more sections

Figures (4)

  • Figure 1: Feedforward and feedback control including anti-windup function.
  • Figure 2: The graph shows the results of our method evaluated on the electric truck example, where the final time $T_f$ has been varied. The red line represents the speed limit, and the gray line the slope. The dashed lines show the reference velocities generated by the optimization, and the corresponding colored full line shows the observed behavior of the truck using that reference.
  • Figure 3: The first graph shows the results of our method evaluated on the electric truck example when the estimated model did not include slope dynamics, where the dashed blue line represents the optimized reference trajectory, the solid blue line the realized trajectory, the red line the speed limit, and the purple line shows the slope of the road throughout the path. The second graph shows the total control input to the system, the blue line, and the feedback control signal in the control loop, the red line.
  • Figure 5: A comparison of our method to TD3 and SAC. Both benchmark DRL policies are the best out of 5 training repetitions spanning 500k timesteps each. All three methods arrive at relatively similar control behavior.