Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PC-Gym: Benchmark Environments For Process Control Problems

Maximilian Bloor, José Torraca, Ilya Orson Sandoval, Akhil Ahmed, Martha White, Mehmet Mercangöz, Calvin Tsay, Ehecatl Antonio Del Rio Chanona, Max Mowbray

TL;DR

PC-Gym introduces a modular, open-source benchmarking suite for chemical process control that enables rigorous evaluation of RL policies against a nonlinear MPC (NMPC) oracle under realistic disturbances and constraints. The framework supports multiple industrially relevant environments (CSTR, Multistage Extraction Column, Crystallization, Four-Tank), three RL algorithms (SAC, DDPG, PPO), and comprehensive performance metrics including optimality gap, dispersion (MAD), and constraint-violation measures. Empirical results across environments reveal that no single RL method dominates; SAC often offers a balance of performance and stability, while DDPG and PPO excel in specific tasks, highlighting the value of task-driven algorithm selection. By coupling standardized environments, disturbance and constraint tooling, and reproducible evaluation, PC-Gym aims to accelerate practical RL-based process control research and its transfer to industry.

Abstract

PC-Gym is an open-source tool for developing and evaluating reinforcement learning (RL) algorithms in chemical process control. It features environments that simulate various chemical processes, incorporating nonlinear dynamics, disturbances, and constraints. The tool includes customizable constraint handling, disturbance generation, reward function design, and enables comparison of RL algorithms against Nonlinear Model Predictive Control (NMPC) across different scenarios. Case studies demonstrate the framework's effectiveness in evaluating RL approaches for systems like continuously stirred tank reactors, multistage extraction processes, and crystallization reactors. The results reveal performance gaps between RL algorithms and NMPC oracles, highlighting areas for improvement and enabling benchmarking. By providing a standardized platform, PC-Gym aims to accelerate research at the intersection of machine learning, control, and process systems engineering. By connecting theoretical RL advances with practical industrial process control applications, offering researchers a tool for exploring data-driven control solutions.

PC-Gym: Benchmark Environments For Process Control Problems

TL;DR

PC-Gym introduces a modular, open-source benchmarking suite for chemical process control that enables rigorous evaluation of RL policies against a nonlinear MPC (NMPC) oracle under realistic disturbances and constraints. The framework supports multiple industrially relevant environments (CSTR, Multistage Extraction Column, Crystallization, Four-Tank), three RL algorithms (SAC, DDPG, PPO), and comprehensive performance metrics including optimality gap, dispersion (MAD), and constraint-violation measures. Empirical results across environments reveal that no single RL method dominates; SAC often offers a balance of performance and stability, while DDPG and PPO excel in specific tasks, highlighting the value of task-driven algorithm selection. By coupling standardized environments, disturbance and constraint tooling, and reproducible evaluation, PC-Gym aims to accelerate practical RL-based process control research and its transfer to industry.

Abstract

PC-Gym is an open-source tool for developing and evaluating reinforcement learning (RL) algorithms in chemical process control. It features environments that simulate various chemical processes, incorporating nonlinear dynamics, disturbances, and constraints. The tool includes customizable constraint handling, disturbance generation, reward function design, and enables comparison of RL algorithms against Nonlinear Model Predictive Control (NMPC) across different scenarios. Case studies demonstrate the framework's effectiveness in evaluating RL approaches for systems like continuously stirred tank reactors, multistage extraction processes, and crystallization reactors. The results reveal performance gaps between RL algorithms and NMPC oracles, highlighting areas for improvement and enabling benchmarking. By providing a standardized platform, PC-Gym aims to accelerate research at the intersection of machine learning, control, and process systems engineering. By connecting theoretical RL advances with practical industrial process control applications, offering researchers a tool for exploring data-driven control solutions.

Paper Structure

This paper contains 38 sections, 32 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Contributions
  4. Background
  5. Dynamical systems
  6. Learning-based Control
  7. Reinforcement Learning
  8. Soft Actor-Critic (SAC)
  9. Deep Deterministic Policy Gradient (DDPG)
  10. Proximal Policy Optimization (PPO)
  11. Constrained Reinforcement Learning
  12. Policy Performance and Reproducibility Measures
  13. Nonlinear MPC (Oracle)
  14. Software Implementation
  15. Control Problems
  16. ...and 23 more sections

Figures (12)

  • Figure 1: The reinforcement learning framework
  • Figure 2: PC-Gym's Architecture
  • Figure 3: Comparison of reinforcement learning algorithms (SAC, PPO, DDPG) and an oracle for control of the CSTR. Controlled variable response, manipulated variable trajectory, and cumulative reward distribution. Shaded areas show min-max ranges across simulations; solid lines represent median performance.
  • Figure 4: Comparison of reinforcement learning algorithms (SAC, PPO, DDPG) and an oracle for Multistage Extraction Column control. Controlled variable response, manipulated variable trajectory. Shaded areas show min-max ranges across simulations; solid lines represent median performance.
  • Figure 5: Cumulative reward distributions (100 repetitions) of reinforcement learning algorithms (SAC, PPO, DDPG) and an oracle for Multistage Extraction Column control.
  • ...and 7 more figures