LoLA: Long Horizon Latent Action Learning for General Robot Manipulation

TL;DR

LoLA tackles the problem of long-horizon language-guided robot manipulation by integrating long-term multi-view observations and proprioception. It introduces SALR, a State-Aware Latent Re-representation that grounds Vision-Language Model embeddings in a physically plausible latent space via a parallel State Transformer and multiplicative fusion, enhanced by learnable masks. An Action Expert based on Conditional Flow Matching decodes multi-step actions from the grounded latent space. Across SIMPLER, LIBERO, and real-world platforms (Franka and BusyBox Aloha), LoLA achieves state-of-the-art performance, especially on long-horizon tasks, demonstrating strong generalization and robustness; limitations remain in extremely perturbation-rich scenarios, with future work focusing on dynamic closed-loop recovery.

Abstract

The capability of performing long-horizon, language-guided robotic manipulation tasks critically relies on leveraging historical information and generating coherent action sequences. However, such capabilities are often overlooked by existing Vision-Language-Action (VLA) models. To solve this challenge, we propose LoLA (Long Horizon Latent Action Learning), a framework designed for robot manipulation that integrates long-term multi-view observations and robot proprioception to enable multi-step reasoning and action generation. We first employ Vision-Language Models to encode rich contextual features from historical sequences and multi-view observations. We further introduces a key module, State-Aware Latent Re-representation, which transforms visual inputs and language commands into actionable robot motion space. Unlike existing VLA approaches that merely concatenate robot proprioception (e.g., joint angles) with VL embeddings, this module leverages such robot states to explicitly ground VL representations in physical scale through a learnable "embodiment-anchored" latent space. We trained LoLA on diverse robotic pre-training datasets and conducted extensive evaluations on simulation benchmarks (SIMPLER and LIBERO), as well as two real-world tasks on Franka and Bi-Manual Aloha robots. Results show that LoLA significantly outperforms prior state-of-the-art methods (e.g., pi0), particularly in long-horizon manipulation tasks.

Figures (4)

• Figure 1: Overview of the proposed model LoLA, which is able to process long-horizon history frames. To align vision-language embeddings with robot actions, the State-Aware Latent Re-representation (SALR) explicitly grounds vision-language features in robot proprioception, creating an "embodiment-anchored" latent space conditioned for action generation.
• Figure 2: Illustration of the State-Aware Latent Re-representation. The latent space $\textbf{(V, S, H)}$ is formed by outer products between the state embeddings and the key-value from the vision-language embeddings, with a hidden dimension of $h$. The dotted arrows denote projection pairs that map visual observations of robot grippers to real-world physical scales (e.g., translation & rotation values).
• Figure 3: Real-world experimental setup featuring Franka Research 3 and bi-manual Aloha robots, displaying multi-view camera inputs and corresponding text instructions. Specifically, Tasks 1, 2, and 3 form a sequential, long-horizon manipulation task: "Put the flat-bottomed pan from the table onto the oven platform".
• Figure 4: Visualization of Representative Task Scenarios.Top: Six fine-grained atomic sub-tasks in the Franka long-horizon cooking task organized into 2 sequential long-horizon tasks, Set up the oven and Remove the pan from the oven and place it back on the table. Bottom: Three distinct tasks from the BusyBox benchmark.