To Switch or Not to Switch? Balanced Policy Switching in Offline Reinforcement Learning

TL;DR

The paper tackles offline reinforcement learning with a one-shot policy switch that incurs a switching cost. It introduces net value and net Q-functions to quantify the trade-off between future return and switching cost, and defines a transport-based switching cost that decomposes into learning and transaction components via optimal transport. A Net Actor-Critic (NAC) algorithm is proposed to search for switch-improving policies using pessimistic offline evaluation and state-dependent policy improvement, followed by a principled switch decision. Empirical results on Gymnasium robotics benchmarks and SUMO-RL traffic control demonstrate meaningful net-value gains for suboptimal old policies, high switching rationality when beneficial, and robustness to cost scaling, highlighting the practical potential of principled policy switching in offline RL.

Abstract

Reinforcement learning (RL) -- finding the optimal behaviour (also referred to as policy) maximizing the collected long-term cumulative reward -- is among the most influential approaches in machine learning with a large number of successful applications. In several decision problems, however, one faces the possibility of policy switching -- changing from the current policy to a new one -- which incurs a non-negligible cost, and in the decision one is limited to using historical data without the availability for further online interaction. Despite the inevitable importance of this offline learning scenario, to our best knowledge, very little effort has been made to tackle the key problem of balancing between the gain and the cost of switching in a flexible and principled way. Leveraging ideas from the area of optimal transport, we initialize the systematic study of policy switching in offline RL. We establish fundamental properties and design a Net Actor-Critic algorithm for the proposed novel switching formulation. Numerical experiments demonstrate the efficiency of our approach on multiple robot control benchmarks of the Gymnasium and traffic light control from SUMO-RL.

Figures (5)

• Figure 1: Comparison between previous setting of online learning and ours.
• Figure 2: Transport switching cost.
• Figure 11: Proposed cost function family. Here we use the shorthands $\mathcal{L}_{ s_i}:=\mathcal{L}(\pi_{\mathrm{o}}(\cdot|s_i), \pi_{\mathrm{n}}(\cdot|s_i))$ and $\mathcal{T}_{s_i}:=\mathcal{T}(\pi_{\mathrm{o}}(\cdot|s_i), \pi_{\mathrm{n}}(\cdot|s_i))$.
• Figure 42: Illustration of the constructed MDP.
• Figure 43: The product action space $\mathcal{A} \times \mathcal{A}$ is partitioned into four components.

Theorems & Definitions (22)

• Definition 3.1: Net Value, Net Q-function
• Definition 3.2: Switch-optimal policy
• Proposition 3.4
• Definition 3.5: Net Bellman operator
• Proposition 3.6: Policy evaluation with net Q-function
• Proposition 3.7: Optimality of the proposed cost
• Definition 1.1: Feasible transport plan
• Definition 1.2: Optimal transport plan
• Proposition 1.3: Feasibility of the proposed transport plan
• Proposition 1.4: Respective optimality of the cost terms