When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoning

TL;DR

This paper analyzes how Group Relative Policy Optimization (GRPO) behaves when trained on multiple domains for reasoning tasks. It demonstrates asymmetric cross-domain transfer, with mathematical reasoning benefiting from other domains, while logic and puzzle show weak transfer. It shows order effects, e.g., math→science yielding higher joint accuracy ($83\%$ math, $41\%$ science) while science→math degrades to ($77\%$, $25\%$); mixed-domain training can stabilize performance. The work provides domain-aware and order-aware training guidance and highlights the need for principled multi-domain GRPO design.

Abstract

Group Relative Policy Optimization (GRPO) has become a key technique for improving reasoning abilities in large language models, yet its behavior under different domain sequencing strategies is poorly understood. In particular, the impact of sequential (one domain at a time) versus mixed-domain (multiple domain at a time) training in GRPO has not been systematically studied. We provide the first systematic analysis of training-order effects across math, science, logic, and puzzle reasoning tasks. We found (1) single-domain generalization is highly asymmetric: training on other domains improves math reasoning by approximately 25\% accuracy, while yielding negligible transfer to logic and puzzle; (2) cross-domain interactions are highly order-dependent: training in the order math$\rightarrow$science achieves 83\% / 41\% accuracy on math / science, while reversing the order to science$\rightarrow$math degrades performance to 77\% / 25\%; (3) no single strategy is universally optimal in multi-domain training: sequential training favors math (up to 84\%), mixed training favors science and logic, and poor ordering can incur large performance gaps (from 70\% to 56\%). Overall, our findings demonstrate that GRPO under multi-domain settings exhibits pronounced asymmetry, order sensitivity, and strategy dependence, highlighting the necessity of domain-aware and order-aware training design.

Figures (10)

• Figure 1: Cross-domain transfer and interference in GRPO.(a) Training-order comparison across domains. Opposite sequential training orders lead to large differences in average multi-domain performance. (b) Single-domain transfer graph. Values (e.g., $+20.8\%$) denote accuracy gains when training on a source domain (science) and evaluating on a target domain (math); darker colors indicate larger gains. Math shows strong cross-domain transfer, science moderate transfer, and logic/puzzle minimal transfer. (c) and (d) Asymmetric two-domain transfer. "$\rightarrow$" means sequential training; “+” denotes mixed training; dashed lines indicate single-domain baselines; points closer to the top-right achieve better joint performance across both domains. Training order is critical: math$\rightarrow$science yields substantially higher accuracy, while science interferes with logic in sequential training.
• Figure 2: Overview of our systematic analysis of GRPO in multi-domain settings. (a) Single-domain generalization: how training on one domain transfers to others. (b) Cross-domain interaction: asymmetric facilitation and interference between two domains under different training orders. (c) Multi-domain training: different training strategies for achieving improved and balanced performance across domains.
• Figure 3: Accuracy on MATH500 at each training step for Qwen3-4B-Base when trained with GRPO on different domain data (math, logic, puzzle and science). Across all domains, training rapidly activates the model’s mathematical reasoning ability.
• Figure 4: Step-wise accuracy of Qwen3-4B-Base on GPQA, Logic Test, and Puzzle Test during GRPO training with data from different domains (a) Results on the science domain (GPQA) show that science reasoning can be partially activated by training on other domains, though less readily than in math. (b) and (c) show results on the logic and puzzle domains respectively. The model’s logic and puzzle reasoning abilities are largely insensitive to training on other domains and are primarily activated by in-domain data..
• Figure 5: Training pipeline for Cross-Domain interaction. We first train the model on Field 1 (math) to obtain $M_1$, then continue training on Field 2 (science) to obtain $M_2$, and finally evaluate Field 1 performance to quantify forgetting/transfer: $\Delta_{\mathrm{F1}\leftarrow \mathrm{F2}}=\mathrm{Acc}_{\mathrm{F1}}(M_2)-\mathrm{Acc}_{\mathrm{F1}}(M_1)$.