HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policy

TL;DR

Vision-Language-Action models traditionally rely solely on the current observation, limiting performance on history-dependent robotic tasks. HAMLET introduces moment tokens to compress per-timestep perceptual information and a lightweight memory module to fuse past context into history-aware conditioning for action prediction, with moment tokens initialized by time-contrastive learning. The framework is plug-and-play and backbone-agnostic, delivering substantial improvements on long-horizon real-world tasks and standard simulation benchmarks without expensive re-training. Empirically, HAMLET achieves notable gains over baselines (e.g., 76.4% average real-world success on GR00T N1.5 with a 47.2% improvement) and enhances performance across RoboCasa Kitchen, LIBERO, and SimplerEnv-Bridge, illustrating practical impact for history-aware robotic manipulation.

Abstract

Inherently, robotic manipulation tasks are history-dependent: leveraging past context could be beneficial. However, most existing Vision-Language-Action models (VLAs) have been designed without considering this aspect, i.e., they rely solely on the current observation, ignoring preceding context. In this paper, we propose HAMLET, a scalable framework to adapt VLAs to attend to the historical context during action prediction. Specifically, we introduce moment tokens that compactly encode perceptual information at each timestep. Their representations are initialized with time-contrastive learning, allowing them to better capture temporally distinctive aspects. Next, we employ a lightweight memory module that integrates the moment tokens across past timesteps into memory features, which are then leveraged for action prediction. Through empirical evaluation, we show that HAMLET successfully transforms a state-of-the-art VLA into a history-aware policy, especially demonstrating significant improvements on long-horizon tasks that require historical context. In particular, on top of GR00T N1.5, HAMLET achieves an average success rate of 76.4% on history-dependent real-world tasks, surpassing the baseline performance by 47.2%. Furthermore, HAMLET pushes prior art performance from 64.1% to 66.4% on RoboCasa Kitchen (100-demo setup) and from 95.6% to 97.7% on LIBERO, highlighting its effectiveness even under generic robot-manipulation benchmarks.

Figures (14)

• Figure 1: Examples of history-dependent long-horizon tasks. (a) Real-world tasks with occlusion (upper) or multi-step reasoning (lower) are inherently non-Markovian: actions cannot be determined from the current view alone. (b) Success rates show that HAMLET clearly surpasses GR00T N1.5 and the naïve multi-frame baseline, effectively leveraging history for reliable long-horizon control.
• Figure 2: An overview of HAMLET. Building on a pre-trained VLA, HAMLET adds two key components: moment tokens, appended to the VLM input and initialized with time-contrastive learning to capture task-relevant representations at each timestep, and a lightweight memory module that aggregates these tokens across timesteps for history-aware action prediction.
• Figure 3: Example rollouts of real-world tasks. We present example rollouts executed by our HAMLET and GR00T N1.5, respectively. While HAMLET predicts proper next actions, in (a) GR00T N1.5 is confused about whether it should lift or release the cube, and in (b) it fails to identify which cup has a cube underneath, due to the absence of historical context.
• Figure 4: What does the memory network memorize? (a) Visualization of self-attention map of moment tokens over input images inside the VLM, showing that they concentrate strongly on task-relevant regions. (b) Normalized self-attention weights of the memory module across the moment token sequence, indicating which timesteps contribute most to the memory features.
• Figure 5: Evaluation environments. Left: Real-world evaluation comprises three tabletop tasks, which especially require awareness on historical context. Right: Simulation benchmarks include RoboCasa nasiriany2024robocasa Kitchen, LIBERO liu2023libero, and SimplerEnv-Bridge li2024evaluating, which consist of diverse indoor manipulation tasks.