Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving a Proportional Integral Controller with Reinforcement Learning on a Throttle Valve Benchmark

Paul Daoudi, Bojan Mavkov, Bogdan Robu, Christophe Prieur, Emmanuel Witrant, Merwan Barlier, Ludovic Dos Santos

TL;DR

This work tackles controlling nonlinear throttle valves with asymmetric hysteresis by blending a fixed discrete-time PI controller with reinforcement learning guided by the PI policy. By restricting the RL search to a vicinity around the PI baseline (PI-RL using Perturbation Action Guided), the method achieves near-optimal performance with substantially reduced data requirements compared with pure RL, as demonstrated across three valves with varying dynamics. Key contributions include the adaptation of Reinforcement Learning with Guides to a throttle-valve setting, the PI-APE and PI-API updates within a restricted action space, and empirical evidence of improved sample efficiency and robust tracking under realistic disturbances. The approach holds practical significance for data-efficient control of nonlinear, stochastic actuators in industrial contexts and invites future work on broader applications and enhanced robustness.",

Abstract

This paper presents a learning-based control strategy for non-linear throttle valves with an asymmetric hysteresis, leading to a near-optimal controller without requiring any prior knowledge about the environment. We start with a carefully tuned Proportional Integrator (PI) controller and exploit the recent advances in Reinforcement Learning (RL) with Guides to improve the closed-loop behavior by learning from the additional interactions with the valve. We test the proposed control method in various scenarios on three different valves, all highlighting the benefits of combining both PI and RL frameworks to improve control performance in non-linear stochastic systems. In all the experimental test cases, the resulting agent has a better sample efficiency than traditional RL agents and outperforms the PI controller.

Improving a Proportional Integral Controller with Reinforcement Learning on a Throttle Valve Benchmark

TL;DR

This work tackles controlling nonlinear throttle valves with asymmetric hysteresis by blending a fixed discrete-time PI controller with reinforcement learning guided by the PI policy. By restricting the RL search to a vicinity around the PI baseline (PI-RL using Perturbation Action Guided), the method achieves near-optimal performance with substantially reduced data requirements compared with pure RL, as demonstrated across three valves with varying dynamics. Key contributions include the adaptation of Reinforcement Learning with Guides to a throttle-valve setting, the PI-APE and PI-API updates within a restricted action space, and empirical evidence of improved sample efficiency and robust tracking under realistic disturbances. The approach holds practical significance for data-efficient control of nonlinear, stochastic actuators in industrial contexts and invites future work on broader applications and enhanced robustness.",

Abstract

This paper presents a learning-based control strategy for non-linear throttle valves with an asymmetric hysteresis, leading to a near-optimal controller without requiring any prior knowledge about the environment. We start with a carefully tuned Proportional Integrator (PI) controller and exploit the recent advances in Reinforcement Learning (RL) with Guides to improve the closed-loop behavior by learning from the additional interactions with the valve. We test the proposed control method in various scenarios on three different valves, all highlighting the benefits of combining both PI and RL frameworks to improve control performance in non-linear stochastic systems. In all the experimental test cases, the resulting agent has a better sample efficiency than traditional RL agents and outperforms the PI controller.
Paper Structure (16 sections, 6 equations, 5 figures, 1 algorithm)

This paper contains 16 sections, 6 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. The throttle valve system
  3. Experimental setup
  4. A non-linear stochastic system
  5. Problem statement and notations
  6. Traditional techniques
  7. Discrete time PI Controller
  8. Approximate Policy Iteration
  9. Combining both methods
  10. Comparison of the different controllers on the experimental test benches
  11. Nominal comparison and reaction to quick reference changes
  12. PI controller
  13. RL-based agents
  14. Robustness to external noise
  15. Sample efficiency
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Experimental test bench: hardware configuration and wiring of the electric throttle valve.
  • Figure 2: Steady state analysis for the $3$ valves.
  • Figure 3: Comparison of agents under different scenarios with access to real outputs. Both figures show the evolution of angles over time with a reference change every $X$ seconds, set to 5 seconds for the left figures and 2.5 seconds for the right figures.
  • Figure 4: Comparison of agents under noisy outputs or controls with varying standard deviations. Each bar represents the mean and standard deviation of the Mean-Squared-Error (MSE) between the reference and the angle across the $3$ valves.
  • Figure 5: Learning curves of the different RL-based agents averaged on the $3$ valves. Agents are both trained with $4$ CPUs during $2500$ episodes, representing $3$ hours.