Reinforcement Learning Constrained Beam Search for Parameter Optimization of Paper Drying Under Flexible Constraints

TL;DR

The paper tackles the challenge of enforcing flexible design constraints in reinforcement learning by introducing Reinforcement Learning Constrained Beam Search (RLCBS), an inference-time decoding strategy that applies constrained beam search to policy-based RL with discrete actions. By leveraging an oracle simulation environment, RLCBS prunes and guides action sequences to satisfy negative and positive constraints while optimizing for energy efficiency in a modular paper-drying testbed. Compared with NSGA-II and a greedy RL baseline, RLCBS achieves competitive energy savings and delivers a significant speed advantage (roughly 2.58x in some settings), demonstrating practical value when design constraints evolve post-training. The approach offers a scalable, constraint-aware decoding paradigm for RL in combinatorial optimization problems where real-time solutions are not mandatory, with potential applicability across process optimization tasks and RL planning domains.

Abstract

Existing approaches to enforcing design constraints in Reinforcement Learning (RL) applications often rely on training-time penalties in the reward function or training/inference-time invalid action masking, but these methods either cannot be modified after training, or are limited in the types of constraints that can be implemented. To address this limitation, we propose Reinforcement Learning Constrained Beam Search (RLCBS) for inference-time refinement in combinatorial optimization problems. This method respects flexible, inference-time constraints that support exclusion of invalid actions and forced inclusion of desired actions, and employs beam search to maximize sequence probability for more sensible constraint incorporation. RLCBS is extensible to RL-based planning and optimization problems that do not require real-time solution, and we apply the method to optimize process parameters for a novel modular testbed for paper drying. An RL agent is trained to minimize energy consumption across varying machine speed levels by generating optimal dryer module and air supply temperature configurations. Our results demonstrate that RLCBS outperforms NSGA-II under complex design constraints on drying module configurations at inference-time, while providing a 2.58-fold or higher speed improvement.

Figures (7)

• Figure 1: Schematic showing one step in RLCBS. We start with $n_b$ beams, each represented by action-state sequence $\{s_0,a_0,... s_{t-1},a_{t-1},s_t\}$. For each beam, the RL policy predicts a discrete probability distribution across the action space $\mathcal{A}$, resulting in $n_b\times|\mathcal{A}|$ logits passed to the logits processor, where invalid actions are masked by setting the corresponding logits to $-\infty$. From the processed logits, two groups of actions are proposed: (1) top-$k$ actions as rated by the RL policy; and (2) actions that will advance the current beam in fulfilling the constraints. Subsequently, $n_b$ actions are selected from the proposed actions with high probabilities while maintaining a balanced mix of constraint fulfillment progress across the beam candidates. Finally, the next-actions $a_t$ are passed to the simulation environment to predict next state $s_{t+1}$ which will extend the beams by one action-state pair.
• Figure 2: Section view of the Smart Dryer testbed. The testbed features a reconfigurable drying chamber that accommodates up to 12 dryer modules in a row. Paper samples are placed on a stainless steel mesh-type conveyer belt moving through the chamber at a constant set speed. Each of the 12 dryer positions can be occupied by one of the 4 dryer modules: slot-jet reattachment (SJR) module, perforated plate (PP) module, di-electrophoresis (DEP) module, and solid plate (SP) module. Finally, three IR emitters are installed between dryers 3-4, 6-7, and 9-10, and can be optionally enabled to assist in drying.
• Figure 3: Sample paper temperature and dry-basis moisture content (DBMC) trajectory as simulated by the physics-based drying model using dryer configuration: 6x SJR, 6x SP and process parameters: total drying time 2.2min, machine speed $v_m = 0.0482m \per s$, hot air supply temperature $T_a = 140.20\celsius$, air velocity $v_a = 11.41m \per s$, IR temperature $T_{\textrm{IR}_1} = 501.99\celsius$, $T_{\textrm{IR}_2} = 677.08\celsius$, $T_{\textrm{IR}_3} = 760.0\celsius$.
• Figure 4: Optimized air temperature $T_a$, air flow rate (in SCFM), IR surface temperature (assuming $T_{\textrm{IR}_1} = T_{\textrm{IR}_2} = T_{\textrm{IR}_3}$), and the resulting overall energy consumption $q$ as functions of machine speed given fixed dryer sequence configuration: SJR in positions 1-6, PP in positions 7-12, and IR on/off between dryers 3-4, 6-7, and 9-10.
• Figure 5: Moving average of cumulative episodic reward over past 100 episodes for the PPO RL agent trained on the Smart Dryer simulation environment with randomized initial machine speed levels.