Affordance Field Intervention: Enabling VLAs to Escape Memory Traps in Robotic Manipulation

TL;DR

AFI addresses memory traps in VLA-based robotic manipulation by introducing a 3D Spatial Affordance Field (SAF) as an on-demand plug-in. It detects memory traps through proprioceptive cues, rolls back to a low-cost high-affordance state, and uses SAF-guided waypoint sampling to re-route trajectories, with a SAF scorer selecting the best VLA proposal. The approach is model-agnostic and training-free, avoiding retraining of VLA backbones. Experiments on real robots and LIBERO-Pro show substantial robustness gains across diverse tasks and OOD scenarios, demonstrating improved generalization without additional data collection.

Abstract

Vision-Language-Action (VLA) models have shown great performance in robotic manipulation by mapping visual observations and language instructions directly to actions. However, they remain brittle under distribution shifts: when test scenarios change, VLAs often reproduce memorized trajectories instead of adapting to the updated scene, which is a failure mode we refer to as the "Memory Trap". This limitation stems from the end-to-end design, which lacks explicit 3D spatial reasoning and prevents reliable identification of actionable regions in unfamiliar environments. To compensate for this missing spatial understanding, 3D Spatial Affordance Fields (SAFs) can provide a geometric representation that highlights where interactions are physically feasible, offering explicit cues about regions the robot should approach or avoid. We therefore introduce Affordance Field Intervention (AFI), a lightweight hybrid framework that uses SAFs as an on-demand plug-in to guide VLA behavior. Our system detects memory traps through proprioception, repositions the robot to recent high-affordance regions, and proposes affordance-driven waypoints that anchor VLA-generated actions. A SAF-based scorer then selects trajectories with the highest cumulative affordance. Extensive experiments demonstrate that our method achieves an average improvement of 23.5% across different VLA backbones ($π_{0}$ and $π_{0.5}$) under out-of-distribution scenarios on real-world robotic platforms, and 20.2% on the LIBERO-Pro benchmark, validating its effectiveness in enhancing VLA robustness to distribution shifts.

Figures (7)

• Figure 1: Memory Trap in VLAs. VLAs often fail under distribution shifts. In such a case, they replay trajectories memorized during training instead of adapting to the updated scene. In both Case 1 and Case 2, even though the target object moves, the VLA drives the end-effector toward the original location, ignoring new spatial cues.
• Figure 2: Spatial Affordance Field (SAF) Construction. (a) GPT-4o decomposes the task instruction into sequential stages and identifies the current target object (e.g., "carrot" or "blue pan"). (b) The target text is fed to Grounded-SAM for segmentation, and the resulting 2D mask is back-projected into 3D space to construct the SAF, where color gradients indicate affordance values.
• Figure 3: Overview of Affordance Field Intervention (AFI). (1) Memory Trap Detection: SAF evaluates VLA-predicted actions and detects memory traps by monitoring end-effector velocity and distance to target. (2) Trajectory Rollback: Upon detection, the robot rolls back to the historical position with lowest SAF cost before grasping attempts. (3) SAF-Guided Sampling: VLA generates trajectory candidates at SAF-sampled waypoints, and the trajectory with lowest cumulative SAF cost is selected for execution.
• Figure 4: Real-world AFI execution rollout. Top: Memory trap detected at $t=50$ when approaching wrong location, followed by rollback to low-cost historical position. Bottom: SAF-guided sampling ($t=70$-$79$) generates trajectory candidates; optimal trajectory (green) is selected and executed ($t=80$-$90$) for successful completion.
• Figure 5: Visualization of object position perturbations in LIBERO simulation, where the target object "black bowl on the cookie box" is displaced to significantly deviated positions.