Local-Global Learning of Interpretable Control Policies: The Interface between MPC and Reinforcement Learning

TL;DR

This paper proposes a local-global paradigm for optimal control that unites model-based optimization (MPC) with data-driven reinforcement learning. It formalizes how a learnable, optimization-based value function (Q^MPCφ) can approximate the global Bellman optimality condition while enabling fast, local online decisions via MPC, and it discusses two integration strategies: modular value-function augmentation and all-in-one learning of MPC components. Through theoretical framing and case studies, the work assesses the benefits and trade-offs of interpretability, safety, and sample efficiency, and highlights practical challenges in online optimization and exploration. The results point to promising directions for safe, near-optimal decision-making under uncertainty by leveraging the complementary strengths of MPC and RL in a local-global framework.

Abstract

Making optimal decisions under uncertainty is a shared problem among distinct fields. While optimal control is commonly studied in the framework of dynamic programming, it is approached with differing perspectives of the Bellman optimality condition. In one perspective, the Bellman equation is used to derive a global optimality condition useful for iterative learning of control policies through interactions with an environment. Alternatively, the Bellman equation is also widely adopted to derive tractable optimization-based control policies that satisfy a local notion of optimality. By leveraging ideas from the two perspectives, we present a local-global paradigm for optimal control suited for learning interpretable local decision makers that approximately satisfy the global Bellman equation. The benefits and practical complications in local-global learning are discussed. These aspects are exemplified through case studies, which give an overview of two distinct strategies for unifying reinforcement learning and model predictive control. We discuss the challenges and trade-offs in these local-global strategies, towards highlighting future research opportunities for safe and optimal decision-making under uncertainty.

Figures (4)

• Figure 1: The local agent's interactions with the environment provide useful feedback towards improving the global MDP solution. The global $Q$-function should satisfy the Bellman optimality condition for all state-action pairs in $\mathcal{S} \times \mathcal{A}$. However, feedback at any given $(s,a)$ requires the local agent to visit these regions, using its policy $\pi$, as represented by the fragment of its trajectory $(s,a,r,s')$. Thus, improvements to the global solution are driven via local actions.
• Figure 2: An MPC-based function approximator is well-aligned with the local-global interface. Top: The local agent designs a coordinated sequence of actions to reach the target state, executing $u_{0|t}^{\star}$ as action $a$. As opposed to hastily taking direct action towards the target state, the agent moves away from the target to later satisfy the state constraints (dashed in red). Bottom: In the global sense, a rich MPC parameterization can effectively approximate the optimal $Q$-function $Q^\star$. In theory, this approximation can be obtained by passing the global agent through the contraction mapping $\mathcal{T}$. However, $\mathcal{T}$ can only be approximately evaluated using observations $(s,a,r,s')$ obtained from the local agent's interactions.
• Figure 3: Closed-loop state profiles of a continuous stirred tank reactor for three optimal decision-making agents: RL, MPC, and RL+MPC lawrence2025view. The MPC agent satisfies the constraints (shaded area), but never achieving the control target (dashed $c_B$ value). The RL agent immediately violates the constraints, but ultimately reaches the control target. The RL+MPC (value function-augmented MPC) agent tames the RL agent's trajectory, satisfying the constraints, while also eventually reaching the target. The state variables are concentrations $c_A$ and $c_B$, as well as reactor and coolant temperatures $T_R$ and $T_K$.
• Figure 4: Episodic adaptation of the model of an MPC agent used as a function approximator banker_gradient-based_2025. Top: Episodic return of MPC as a policy function approximator with learnable state-space model matrices $A$ and $B$. Initially, the observations of objective $J(\theta)$ exhibit large variance. After several learning episodes, the agent approaches the optimal $J(\theta^\star)$. Bottom: Mismatch between the MPC agent's state-space model matrices and those of the true system (denoted by $M_\phi$ and $M$, respectively) quantified in terms of the Frobenius norm $||\cdot||_{F}$. Notably, as the agent's performance improves, the model mismatch increases. Lines represent the mean of 100 runs with $\pm 2$ sample standard deviations shaded.