HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agents

TL;DR

This work proposes HiMAC, a hierarchical agentic RL framework that explicitly decomposes long-horizon decision-making into macro-level planning and micro-level execution and introduces a critic-free hierarchical policy optimization paradigm that extends group-based reinforcement learning to bi-level structures through hierarchical relative advantage estimation.

Abstract

Large language model (LLM) agents have recently demonstrated strong capabilities in interactive decision-making, yet they remain fundamentally limited in long-horizon tasks that require structured planning and reliable execution. Existing approaches predominantly rely on flat autoregressive policies, where high-level reasoning and low-level actions are generated within a single token sequence, leading to inefficient exploration and severe error propagation over extended trajectories. In this work, we propose HiMAC, a hierarchical agentic RL framework that explicitly decomposes long-horizon decision-making into macro-level planning and micro-level execution. HiMAC models reasoning as a structured blueprint generation process followed by goal-conditioned action execution, enabling robust long-horizon planning within LLM-based agents. To train this hierarchy efficiently, we introduce a critic-free hierarchical policy optimization paradigm that extends group-based reinforcement learning to bi-level structures through hierarchical relative advantage estimation. Furthermore, we propose an iterative co-evolution training strategy that alternates between planner exploration and executor adaptation, mitigating the non-stationarity inherent in hierarchical learning. Extensive experiments on ALFWorld, WebShop, and Sokoban demonstrate that HiMAC consistently outperforms strong prompting and reinforcement learning baselines, achieving state-of-the-art performance and substantially improved sample efficiency across both text-based and visually grounded environments. Our results show that introducing structured hierarchy, rather than increasing model scale alone, is a key factor for enabling robust long-horizon agentic intelligence.

Figures (3)

• Figure 1: A typical case comparsion in the WebShop environment and overall performance. (a) Flat-policy architectures easily get lost in irrelevant observations (Context Drift). (b) HiMAC addresses this exploration inefficiency through a bi-level decoupled architecture (Planner and Executor sharing the same $\pi_{\theta}$), mapping structured blueprints to executable atomic actions. (c) HiMAC yields consistent state-of-the-art performance against advanced RL reasoning baselines across multiple environments.
• Figure 2: Overall pipeline of HiMAC. Phase A (left) optimizes the macro-policy for blueprint generation using GRPO while freezing the micro-policy. Phase B (right) optimizes the micro-policy for atomic action execution conditioned on a fixed blueprint $z^*$. Both phases update the same underlying LLM parameters $\theta$.
• Figure 3: Evolution of HiMAC's planning and emergent behaviors. For the task "put a candle in toilet": (Left) Early training shows blind exploration due to lacking spatial priors. (Right) Post-convergence, the planner efficiently navigates to the target and develops emergent self-verification.