Control of dynamical systems with neural networks

TL;DR

The paper explores neural-network–based control of dynamical systems, parameterizing inputs with neural controllers for discrete- and continuous-time dynamics under deterministic and stochastic settings. It introduces NODEC and NNC frameworks, demonstrates near-optimal control across diverse domains (ecology, inventory, microbiology, oscillator networks), and contrasts neural-control with model predictive control. A key contribution is showing that gradient-based training of neural controllers can closely approximate optimal policies, often with substantial computational advantages, plus integrating conformal prediction to quantify uncertainty. The work highlights practical impact for complex real-world control tasks and provides code resources to foster further adoption and extension.

Abstract

Control problems frequently arise in scientific and industrial applications, where the objective is to steer a dynamical system from an initial state to a desired target state. Recent advances in deep learning and automatic differentiation have made applying these methods to control problems increasingly practical. In this paper, we examine the use of neural networks and modern machine-learning libraries to parameterize control inputs across discrete-time and continuous-time systems, as well as deterministic and stochastic dynamics. We highlight applications in multiple domains, including biology, engineering, physics, and medicine. For continuous-time dynamical systems, neural ordinary differential equations (neural ODEs) offer a useful approach to parameterizing control inputs. For discrete-time systems, we show how custom control-input parameterizations can be implemented and optimized using automatic-differentiation methods. Overall, the methods presented provide practical solutions for control tasks that are computationally demanding or analytically intractable, making them valuable for complex real-world applications.

Figures (14)

• Figure 1: Schematic of a discrete-time control problem, where a control input $\hat{u}_k(\cdot; w)$ is parameterized by a neural network with parameters $w\in\mathbb{R}^N$. We refer to these control inputs as neural network controllers (NNCs). (a) The overall system structure, in which the NNC generates control inputs that influence the state transition function $f(\cdot)$. When the NNC depends on the system state $x_k$, a skip connection (dashed grey line) propagates state information directly from the controller input to $f(\cdot)$, creating a ResNet-like unfolding of the underlying dynamics. (b) An open-loop control scenario, where the control input $\hat{u}_k(w)$ depends on the time step $k$ but not on the system state $x_k$. (c) A closed-loop control scenario, where the control input $\hat{u}_k(x_k; w)$ is computed based on the current state $x_k$.
• Figure 2: Comparison of control approaches for the Beverton--Holt model \ref{['eq:beverton_holt']} with harvesting over a finite time horizon $T = 10$, using parameters $\gamma = 1$, $a = 0.1$, $c = 10$, and $r = 1.5$. The control objective is to maximize the total discounted net benefit over $T$, as defined in Eq. \ref{['eq:bh_j1']}. (a) Evolution of the population $x_k$ under the optimal control (OC) and the NNC approaches. (b) Corresponding control input over time. The population initially increases due to low control, then stabilizes as the control intensifies. The NNC closely resembles the OC solution.
• Figure 3: Predator-prey ABM. Snapshot of a three-species predator-prey ABM simulation on a $51\times 51$ grid. Grid cells colored green and light brown indicate nutrient-rich and nutrient-poor regions, respectively.
• Figure 4: Control of a predator-prey ABM using an NNC. (a) To control the predator-prey ABM, we first define appropriate inputs and outputs for the NNC. Potential inputs include the population sizes $a_k$, $b_k$, and $c_k$ of species $A$, $B$, and $C$ at time step $k$. Since we aim to directly control predator and prey populations, the ANN produces two outputs, $\hat{u}_1$ and $\hat{u}_2$. A problem-specific straight-through estimator is used to obtain integer-valued control actions by removing the fractional part $\{\max\{0,\cdot\}\}$ from the positive hidden-layer outputs. We denote the hidden-layer activations by $\sigma$, and the straight-through estimator by $\{x^+\}$. (b) Evolution of predators, prey, and nutrient-right lattice sites in a single realization of the predator-prey ABM. The vertical dashed gray line marks the time when the NNC is activated. The controller is designed to increase the mean number of prey by 10% and reduce the mean number of predators by 50%. Dashed blue and red lines indicate the target prey and predator levels, respectively (i.e., $\bar{b}^* = 4575$ and $\bar{c}^* = 948$). The simulation uses a $255 \times 255$ grid with initial conditions $b_0 = 2500$, $c_0 = 1250$, and parameters $\alpha_1 = 4.0$, $\alpha_2 = 5.0$, $\kappa_1 = 4.0$, $\kappa_2 = 20.0$, and $\tau = 30$wilensky1997netlogo. Initially, 50% of the lattice sites are nutrient-rich.
• Figure 5: Controlling single-sourcing dynamics using an NNC. (a) Training performance of the learned ordering policy, showing the expected cost per period over training epochs for single-sourcing dynamics with lead time $l = 0$, unit holding cost $h = 5$, unit shortage cost $b = 495$, and demand distribution $\mathcal{U}\{0, 4\}$. The dashed black line indicates the optimal cost of 10 achieved by the base-stock policy. The simulation time was approximately 2 minutes on a standard laptop. (b) Example trajectory of the learned policy, illustrating order quantities (solid black line) and inventory levels (dashed blue line) over time.