LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Tasks

TL;DR

The paper tackles long-horizon embodied tasks by introducing LoHoVLA, a unified Vision–Language–Action model that jointly performs high-level sub-task planning and low-level action control using a single pretrained vision-language backbone. It pairs this with a hierarchical closed-loop control mechanism to manage errors at both planning and execution levels, and trains on LoHoSet, a Ravens-based dataset of 20 long-horizon tasks with 1,000 demonstrations each. Empirical results show LoHoVLA outperforms hierarchical baselines and vanilla VLA models on seen and unseen tasks, demonstrating stronger reasoning, planning, and generalization capabilities. The work highlights the potential of end-to-end unified architectures for robust, generalizable embodied intelligence in complex environments.

Abstract

Real-world embodied agents face long-horizon tasks, characterized by high-level goals demanding multi-step solutions beyond single actions. Successfully navigating these requires both high-level task planning (i.e., decomposing goals into sub-tasks) and low-level motion control (i.e., generating precise robot actions). While existing vision language action (VLA) models and hierarchical architectures offer potential in embodied tasks, the former often falter in planning, and the latter can suffer from coordination issues, both hampering performance. We introduce a new unified VLA framework for long-horizon tasks, dubbed LoHoVLA, to overcome these limitations. LoHoVLA leverages a large pretrained vision language model (VLM) as the backbone to jointly generate language and action tokens for sub-task generation and robot action prediction, respectively. This shared representation promotes better generalization across tasks. Additionally, LoHoVLA embraces a hierarchical closed-loop control mechanism to mitigate errors originating from both high-level planning and low-level control. To train LoHoVLA, we introduce LoHoSet, a dataset built on the Ravens simulator, containing 20 long-horizon tasks, each with 1,000 expert demonstrations composed of visual observations, linguistic goals, sub-tasks, and robot actions. Experimental results show that LoHoVLA significantly surpasses both hierarchical and standard VLA approaches on long-horizon embodied tasks in the Ravens simulator. These findings underscore the promise of unified architectures for advancing generalizable embodied intelligence.

Figures (4)

• Figure 1: Left top: Vanilla VLA directly maps high-level goals and observations to actions. Left bottom: The hierarchical architecture separates planning and execution— the planner infers sub-tasks, and the controller executes them. Right: LoHoVLA integrates high-level task planning and low-level motion control into a unified model. It uses an auto-regressive (AR) Transformer as its backbone and employs a hierarchical closed-loop control mechanism.
• Figure 2: An example of the long-horizon LoHoSet. Object attributes like size, color, quantity, and position vary across cases.
• Figure 3: (a) Comparison of performance on unseen tasks between training with and without dataset expansion, evaluated using the sub-task planning success rate (%). (b) Comparison of performance between one-stage and two-stage training strategies, evaluated using both the sub-task planning success rate (%) and task completion success rate (%).
• Figure 4: Visual examples of all tasks in the LoHoSet dataset, including 3 pick-and-place primitives and 20 long-horizon tasks, showcasing the diversity and complexity of the task scenarios.