B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learning

TL;DR

This work tackles extrapolation error and value overestimation in offline multi-agent RL (MARL) by introducing Behavior Cloning regularization with Critic Clipping (B3C), a minimalist method that augments existing online MARL algorithms. B3C regularizes the policy via BC while stabilizing the critic through clipping of the target value, enabling a higher weight on the RL objective and improving performance. The method is integrated with non-linear value factorization in FACMAC, yielding FACMAC+B3C, and is validated across diverse offline MARL benchmarks, including multi-agent Mujoco and particle environments, with strong results and thorough ablations. The findings highlight practical benefits of critic clipping and the efficacy of non-monotonic factorization in offline MARL, offering a simple, robust approach for data-efficient multi-agent decision-making.

Abstract

Overestimation arising from selecting unseen actions during policy evaluation is a major challenge in offline reinforcement learning (RL). A minimalist approach in the single-agent setting -- adding behavior cloning (BC) regularization to existing online RL algorithms -- has been shown to be effective; however, this approach is understudied in multi-agent settings. In particular, overestimation becomes worse in multi-agent settings due to the presence of multiple actions, resulting in the BC regularization-based approach easily suffering from either over-regularization or critic divergence. To address this, we propose a simple yet effective method, Behavior Cloning regularization with Critic Clipping (B3C), which clips the target critic value in policy evaluation based on the maximum return in the dataset and pushes the limit of the weight on the RL objective over BC regularization, thereby improving performance. Additionally, we leverage existing value factorization techniques, particularly non-linear factorization, which is understudied in offline settings. Integrated with non-linear value factorization, B3C outperforms state-of-the-art algorithms on various offline multi-agent benchmarks.

Figures (7)

• Figure 1: Test return with respect to dataset quality (average return of the dataset) in the Cooperative Navigation (CN) and World environments, reported in shao2024counterfactual. Performance close to $y = x$ indicates a strong dependency on data quality.
• Figure 2: Performance difference between the worst-performing seed of MA-TD3+BC and that of MA-TD3+B3C, with negative values indicating that MA-TD3+BC performs worse than MA-TD3+B3C. A value of $-5\%$ represents that BC performs $5\%$ worse than B3C.
• Figure 3: Test return and target value during policy evaluation in the training of BC (blue) and B3C (red). The black dotted line indicates the moment when the target value starts to diverge.
• Figure 4: Performance with respect to the RL coefficient $\alpha$ in the Halfcheetah environment.
• Figure 5: Ablation study on the clipping value, $M$: Performance of FACMAC+B3C with respect to $M$. $M=\infty$ (no clip) corresponds to FACMAC+BC. The performance of OMIGA is also included as a blue dotted line. Error bars represent one standard deviation.