RoboHiMan: A Hierarchical Evaluation Paradigm for Compositional Generalization in Long-Horizon Manipulation

TL;DR

RoboHiMan tackles the problem of compositional generalization in long-horizon robotic manipulation under diverse perturbations. It introduces HiMan-Bench and a hierarchical evaluation paradigm to systematically dissect planning and execution in multi-stage tasks. Key contributions include four task types (A, AP, C, CP), progressive training data layers L1-L4, and vanilla/decoupled/coupled evaluation modes, revealing persistent gaps in current Vision-Language-Action approaches. The work demonstrates that mere data scaling is insufficient for compositional generalization and highlights perturbation robustness and planner-execution coordination as critical bottlenecks, with real-world validation showing tangible benefits of hierarchical guidance.

Abstract

Enabling robots to flexibly schedule and compose learned skills for novel long-horizon manipulation under diverse perturbations remains a core challenge. Early explorations with end-to-end VLA models show limited success, as these models struggle to generalize beyond the training distribution. Hierarchical approaches, where high-level planners generate subgoals for low-level policies, bring certain improvements but still suffer under complex perturbations, revealing limited capability in skill composition. However, existing benchmarks primarily emphasize task completion in long-horizon settings, offering little insight into compositional generalization, robustness, and the interplay between planning and execution. To systematically investigate these gaps, we propose RoboHiMan, a hierarchical evaluation paradigm for compositional generalization in long-horizon manipulation. RoboHiMan introduces HiMan-Bench, a benchmark of atomic and compositional tasks under diverse perturbations, supported by a multi-level training dataset for analyzing progressive data scaling, and proposes three evaluation paradigms (vanilla, decoupled, coupled) that probe the necessity of skill composition and reveal bottlenecks in hierarchical architectures. Experiments highlight clear capability gaps across representative models and architectures, pointing to directions for advancing models better suited to real-world long-horizon manipulation tasks. Videos and open-source code can be found on our project website: https://chenyt31.github.io/robo-himan.github.io/.

Figures (11)

• Figure 1: RoboHiMan Overview. To evaluate compositional generalization, RoboHiMan introduces: (a) HiMan-Bench with four task types: atomic (A), atomic-perturbation (AP), compositional (C), and compositional-perturbation (CP); (b) a hierarchical evaluation paradigm with diverse metrics and progressive training data (L1–L4), where L1 uses minimal atomic data and L4 provides larger datasets; (c) Extensive experiments highlight critical performance gaps across training datasets and evaluation modes, often overlooked by prior benchmarks (notation "X $\rightarrow$ Y" denoting training on Level X and evaluation on task category Y).
• Figure 2: This figure illustrates the key design of HiMan-Bench, including (1) HiMan-Bench task distribution, and (2) multi-level training dataset design in HiMan-Bench.
• Figure 3: The overview of hierarchical evaluation paradigm
• Figure 4: Performance scaling curves on atomic and compositional tasks under different scaling levels.
• Figure 5: Robustness under perturbations across different settings.