Partial Action Replacement: Tackling Distribution Shift in Offline MARL

TL;DR

Offline MARL suffers from distribution shift due to unseen joint actions. The paper introduces partial action replacement (PAR) under factorized behavior policies and SPaCQL, an adaptive mixture of partial backups weighted by ensemble uncertainty, to mitigate OOD updates; it proves that distribution shift scales linearly with the number of deviating agents, i.e. $W_1(d^{(S)}, d^{(\varnothing)}) \le \frac{\gamma}{1-\gamma} \sum_{i\in S} \mathrm{TV}(\pi_i, \mu_i)$, and provides a MARL-specific value-estimation bound. SPaCQL adaptively balances single-agent and coordinated backups via uncertainty-informed weights, yielding a provably tighter bound when data coverage favors localized updates. Empirically, SPaCQL outperforms baselines on many tasks, especially under Random and Medium-Replay datasets, demonstrating robustness to data quality and coordination structure; on Expert datasets performance is competitive, highlighting the method’s ability to adjust to data properties. Overall, the work offers a principled, uncertainty-aware framework for safe offline MARL with reduced reliance on full joint-action exploration and improved sample efficiency in realistic data regimes.

Abstract

Offline multi-agent reinforcement learning (MARL) is severely hampered by the challenge of evaluating out-of-distribution (OOD) joint actions. Our core finding is that when the behavior policy is factorized - a common scenario where agents act fully or partially independently during data collection - a strategy of partial action replacement (PAR) can significantly mitigate this challenge. PAR updates a single or part of agents' actions while the others remain fixed to the behavioral data, reducing distribution shift compared to full joint-action updates. Based on this insight, we develop Soft-Partial Conservative Q-Learning (SPaCQL), using PAR to mitigate OOD issue and dynamically weighting different PAR strategies based on the uncertainty of value estimation. We provide a rigorous theoretical foundation for this approach, proving that under factorized behavior policies, the induced distribution shift scales linearly with the number of deviating agents rather than exponentially with the joint-action space. This yields a provably tighter value error bound for this important class of offline MARL problems. Our theoretical results also indicate that SPaCQL adaptively addresses distribution shift using uncertainty-informed weights. Our empirical results demonstrate SPaCQL enables more effective policy learning, and manifest its remarkable superiority over baseline algorithms when the offline dataset exhibits the independence structure.

Figures (3)

• Figure 1: Action distributions in different dataset types. Blue, yellow and green lines show the range of agent actions, with $a_1^\tau$ denoting agent 1's action from the dataset. (a) Random datasets: Agents act independently, so agent 2's actions may span the full range regardless of agent 1's choice. Partial action replacement (fixing agent 1) allows reasonable coverage of agent 2's action space. (b) Expert dataset with tight correlation: Agents' actions are strongly correlated within a narrow band. Fixing agent 1 may still lead to severe OOD issue. (c) Expert dataset with loose correlation: While correlated, actions have sufficient spread. Fixing agent 1 still allows reasonable coverage of agent 2's action space. This illustrates the potential advantage of partial action replacement for random datasets and some expert datasets, but it may struggle with highly coordinated behaviors.
• Figure 2: Comparison of the uncertainty of Q-value estimation between CFCQL and ICQL-QS on the Cooperative Navigation benchmark. Uncertainty is measured as the standard deviation of Q-estimates from an ensemble of networks. ICQL-QS, which uses partial action replacement yields significantly lower estimation uncertainty on Random dataset consistently. On Expert dataset, the two methods show similar estimation uncertainty, but finally ICQL-QS still leads to lower uncertainty.
• Figure 3: Values of weights with min-max normalization.

Theorems & Definitions (12)

• Lemma 1: Linear Divergence Bound
• Theorem 1: Tight Value-Error Bound
• Definition 1: Maximal Excess Correlation
• Lemma 2: Linear Divergence with Correlations
• Theorem 2: Value-Error Bound with Correlations
• proof
• Theorem 3: SPaCQL Value-Error Bound
• proof : Proof Sketch
• Proposition 1: Gradient equivalence of ICQL-QS
• proof