HiF-VLA: Hindsight, Insight and Foresight through Motion Representation for Vision-Language-Action Models

TL;DR

HiF-VLA addresses temporal myopia in Vision-Language-Action models by introducing motion-based history and foresight, encoded as compact Motion Vectors. It jointly reasons hindsight, insight, and foresight through a Hindsight-Modulated Joint Expert, enabling a think-while-acting paradigm for long-horizon manipulation. The approach achieves state-of-the-art performance on LIBERO-Long and CALVIN-ABC-D with low latency and demonstrates strong real-world applicability on a robotic platform. Limitations include sensitivity to motion estimation noise and opportunities for richer 3D representations and larger-scale pretraining in future work.

Abstract

Vision-Language-Action (VLA) models have recently enabled robotic manipulation by grounding visual and linguistic cues into actions. However, most VLAs assume the Markov property, relying only on the current observation and thus suffering from temporal myopia that degrades long-horizon coherence. In this work, we view motion as a more compact and informative representation of temporal context and world dynamics, capturing inter-state changes while filtering static pixel-level noise. Building on this idea, we propose HiF-VLA (Hindsight, Insight, and Foresight for VLAs), a unified framework that leverages motion for bidirectional temporal reasoning. HiF-VLA encodes past dynamics through hindsight priors, anticipates future motion via foresight reasoning, and integrates both through a hindsight-modulated joint expert to enable a ''think-while-acting'' paradigm for long-horizon manipulation. As a result, HiF-VLA surpasses strong baselines on LIBERO-Long and CALVIN ABC-D benchmarks, while incurring negligible additional inference latency. Furthermore, HiF-VLA achieves substantial improvements in real-world long-horizon manipulation tasks, demonstrating its broad effectiveness in practical robotic settings.

Figures (10)

• Figure 1: (a-b) Comparison with existing methods: VLAs rely on instantaneous observations kim2024openvlakim2025fine (a-top), stack multiple past frames liu2025towardstianpredictive (a-second), or generate pixel-level subgoals zhao2025cotzhang2025up (a-third), suffering from redundancy, high inference cost, and weak structure. In contrast, HiF-VLA (a-bottom) jointly models Hindsight, Insight, and Foresight, expanding the temporal receptive field bidirectionally for compact, structured, and efficient reasoning. (c) HiF-VLA reduces inference latency and achieves state-of-the-art performance on LIBERO-Long and CALVIN ABC-D, significantly outperforming the baseline in real-world experiments.
• Figure 2: HiF-VLA Pipeline. (a) In Hindsight Prior Acquisition (see \ref{['sec:hindsight']}), HiF-VLA encodes dense historical frame sequences into compact Motion Vector (MV) streams, forming structured hindsight primitives that capture temporal dynamics without pixel redundancy. (b) In Foresight Reasoning with Insight (see \ref{['sec:foresight']}), the VLM interprets the task instruction and current observation to infer plausible foresight motions and corresponding latent action tokens. (c) Finally, the Hindsight-Modulated Joint Expert (see \ref{['sec:joint']}) fuses hindsight, foresight, and action representations within a unified latent space, producing temporally consistent and causally coherent action predictions.
• Figure 3: Effect of hindsight length on performance and efficiency. (a) Example of historical frames. (b) HiF-VLA maintains low inference latency as hindsight length increases. (c) Performance of hindsight of different lengths in third-view and multi-view perspectives.
• Figure 4: Performance comparison on different hindsight embedding locations. (a) represents direct injection into the VLM, and (b) represents conditional embedding as an expert decoder. (c) shows the performance of both on LIBERO-Long. $M_h$ denotes the hindsight tokens, $M_f$ represents the foresight tokens and $A_f$ is the action tokens.
• Figure 5: Real-world long-horizon tasks. (a) We deploy our system on the AgileX Piper robotic arm equipped with an external scene camera (Intel RealSense D435) and a wrist-mounted camera. (b) We design three long-horizon tasks covering diverse primitives such as pick, put, cover, stack, and press, emphasizing temporal consistency in action generation.