Soft Conflict-Resolution Decision Transformer for Offline Multi-Task Reinforcement Learning

TL;DR

The paper tackles gradient conflicts in offline multi-task reinforcement learning by challenging coarse mask strategies. It introduces SoCo-DT, a soft conflict-resolution approach that leverages Fisher-information to assign per-parameter importance and uses an adaptive, IQR-based sparsity scheme with an asymmetric cosine annealing schedule to evolve masks during training. The key contributions are the soft importance-aware masking mechanism and TAMU (Task-Aware Mask Update with Adaptive Sparsity), which together enable finer-grained subspace sharing and stable learning across diverse tasks. Empirical results on Meta-World show SoCo-DT achieving state-of-the-art improvements, particularly on sub-optimal datasets, underscoring its practical impact for robust offline multi-task reinforcement learning.

Abstract

Multi-task reinforcement learning (MTRL) seeks to learn a unified policy for diverse tasks, but often suffers from gradient conflicts across tasks. Existing masking-based methods attempt to mitigate such conflicts by assigning task-specific parameter masks. However, our empirical study shows that coarse-grained binary masks have the problem of over-suppressing key conflicting parameters, hindering knowledge sharing across tasks. Moreover, different tasks exhibit varying conflict levels, yet existing methods use a one-size-fits-all fixed sparsity strategy to keep training stability and performance, which proves inadequate. These limitations hinder the model's generalization and learning efficiency. To address these issues, we propose SoCo-DT, a Soft Conflict-resolution method based by parameter importance. By leveraging Fisher information, mask values are dynamically adjusted to retain important parameters while suppressing conflicting ones. In addition, we introduce a dynamic sparsity adjustment strategy based on the Interquartile Range (IQR), which constructs task-specific thresholding schemes using the distribution of conflict and harmony scores during training. To enable adaptive sparsity evolution throughout training, we further incorporate an asymmetric cosine annealing schedule to continuously update the threshold. Experimental results on the Meta-World benchmark show that SoCo-DT outperforms the state-of-the-art method by 7.6% on MT50 and by 10.5% on the suboptimal dataset, demonstrating its effectiveness in mitigating gradient conflicts and improving overall multi-task performance.

Figures (5)

• Figure 1: Average success rate of four conflict-handling strategies on MT15: ① PromptDT (no mask), ② HarmoDT (hard mask, SOTA), ③ Ours (soft mask, ideal). (a) Meta-World with near-optimal datasets; (b) Meta-World with sub-optimal datasets. See §\ref{['sec:experiments']} for details.
• Figure 2: (a) Average number of important parameters wrongly masked during training under Fisher-based detection. (b) Distribution of conflicting parameters among tasks.
• Figure 3: Illustrates the overall framework of SoCo-DT. The left panel shows the process of identifying task-optimal subspaces using the soft masking strategy and TAMU. The right panel presents the workflow of SoCo-DT based on a prompt-enhanced Decision Transformer architecture.
• Figure 4: Average return comparison for MT5 and MT30 task settings. (a) shows average return curves for MT5 and its sub-task variant; (b) shows the same for MT30. Both are compared with the state-of-the-art method HarmoDT-F.
• Figure 5: Ablation study under the near-optimal setting on the MT30. Default values: initial sparsity = 0.2, $\alpha = 20$, $\beta_{\text{left\_max}}$ = 20, $\beta_{\min}$ = 5, $\beta_{\text{right\_max}}$ = 30, $q_1$ = 0.05, $q_3$ = 0.95.