HiMA-Ecom: Enabling Joint Training of Hierarchical Multi-Agent E-commerce Assistants

Abstract

Hierarchical multi-agent systems based on large language models (LLMs) have become a common paradigm for building AI assistants in vertical domains such as e-commerce, where a master agent coordinates multiple specialized sub-agents. Despite their practical importance, realistic benchmarks for training and evaluating such systems remain scarce, and joint optimization across functionally distinct agents is still challenging. To address this gap, we introduce HiMA-Ecom, the first hierarchical multi-agent benchmark tailored for e-commerce scenarios. HiMA-Ecom contains 22.8K instances, including agent-specific supervised fine-tuning samples with memory and system-level input-output pairs for joint multi-agent reinforcement learning. Building upon it, a joint training method named HiMA-R1 is proposed. It presents Variance-Reduction Group Relative Policy Optimization (VR-GRPO), which employs initial trajectory-based Monte Carlo sampling to mitigate the exponential joint action space and selects informative agent groups for efficient updates based on reward variance. Furthermore, an adaptive memory evolution mechanism that repurposes GRPO rewards as cost-free supervisory signals is designed to eliminate repetitive reasoning and accelerate convergence. Experiments on HiMA-Ecom demonstrate that our method, built upon smaller 3B/7B open-source models, achieves performance comparable to that of larger LLMs, such as DeepSeek-R1, and surpasses DeepSeek-V3 by an average of 6\%.

Figures (10)

• Figure 1: Comparison of training paradigms for hierarchical LLM-based multi-agent systems in vertical domains: (a) Master-Only Training, (b) Parameter-Sharing Training, (c) Separate Training for Each Agent with fixed planning, and (d) Joint Multi-Agent RL Training with dynamic planning. The first three settings adopt static invocation structures, while this work focuses on the joint training paradigm with learned invocation in (d), which enables system-level coordination for vertical-domain applications.
• Figure 2: An illustrative example of the hierarchical multi-agent system under e-commerce scenarios. The agent-level reasoning and memory recall are visualized for clarity, while the released dataset consists of agent-specific SFT memory entries and joint input–output pairs for multi-agent RL training, without intermediate supervision.
• Figure 3: VR-GRPO for Multi-Agent Systems (MAS): (1) Initial trajectory-based Monte Carlo sampling to sequentially sample agent actions along an initial trajectory, mitigating action space explosion. (2) Action reward calculation, incorporating accuracy, format, and efficiency rewards based on trajectory output. (3) Marginal benefit-driven updating, which prioritizes the top-$K$ agents with the highest reward variance to accelerate policy evolution.
• Figure 4: Different sampling strategies for multi-agents. Green circles indicate nodes sampling multiple actions via GRPO, while gray circles involve no additional sampling. Compared to naive sampling in (a), (b) could avoid the exponential explosion of the multi-agent joint action space.
• Figure 5: HiMA-R1 with memory achieves higher reward peaks faster during training.