Model-Based Reinforcement Learning Control of Reaction-Diffusion Problems

TL;DR

This work adapts an existing reinforcement learning algorithm using a stochastic policy gradient method and introduces two novel reward functions to drive the flow of the transported field and demonstrates the use of automatic control strategies to initial boundary value problems in thermal and disease transport.

Abstract

Mathematical and computational tools have proven to be reliable in decision-making processes. In recent times, in particular, machine learning-based methods are becoming increasingly popular as advanced support tools. When dealing with control problems, reinforcement learning has been applied to decision-making in several applications, most notably in games. The success of these methods in finding solutions to complex problems motivates the exploration of new areas where they can be employed to overcome current difficulties. In this paper, we explore the use of automatic control strategies to initial boundary value problems in thermal and disease transport. Specifically, in this work, we adapt an existing reinforcement learning algorithm using a stochastic policy gradient method and we introduce two novel reward functions to drive the flow of the transported field. The new model-based framework exploits the interactions between a reaction-diffusion model and the modified agent. The results show that certain controls can be implemented successfully in these applications, although model simplifications had to be assumed.

Figures (12)

• Figure 1: Overview of the RL algorithm adapted from mao_resource_2016.
• Figure 2: Software setup
• Figure 3: $L^2$ norm of infections and diffusivities and reward function value as in eq. (\ref{['eq:R1']}) on the left side and as in eq. (\ref{['eq:R2']}) on the right side from before in orange and after training in blue.
• Figure 4: RL simulation results for state $c$ and diffusivities at start time on the top and at final time on the bottom resulting from choosing reward function eq. (\ref{['eq:R1']}).
• Figure 5: RL simulation results for state $c$ and diffusivities at start time on the top and at final time on the bottom resulting from choosing reward function eq. (\ref{['eq:R2']}).