Learning Affordances at Inference-Time for Vision-Language-Action Models

TL;DR

Learning from Inference-Time Execution (LITEN) tackles long-horizon robotic control by coupling a high-level Vision-Language Model with a low-level VLA, enabling the high-level model to learn robot affordances from inference-time experiences. It uses a two-phase loop—reasoning to plan with in-context past rollouts, and assessment to distill feedback via structured prompts—so that plans improve over repeated attempts without additional training. Experiments on real-world DROID setups show that LITEN progressively enhances planning and task success compared to baselines and ablations, demonstrating the value of in-context experiential learning for robotics. The approach offers a hardware-agnostic, training-free path for aligning semantic plan generation with concrete motor capabilities in open-world environments, with potential for lifelong learning and scalable context management in the future.

Abstract

Solving complex real-world control tasks often takes multiple tries: if we fail at first, we reflect on what went wrong, and change our strategy accordingly to avoid making the same mistake. In robotics, Vision-Language-Action models (VLAs) offer a promising path towards solving complex control tasks, but lack the ability to contextually and dynamically readjust behavior when they fail to accomplish a task. In this work, we introduce Learning from Inference-Time Execution (LITEN), which connects a VLA low-level policy to a high-level VLM that conditions on past experiences by including them in-context, allowing it to learn the affordances and capabilities of the low-level VLA. Our approach iterates between a reasoning phase that generates and executes plans for the low-level VLA, and an assessment phase that reflects on the resulting execution and draws useful conclusions to be included in future reasoning contexts. Unlike similar approaches to self-refinement in non-robotics domains, LITEN must reflect on unstructured real-world robot trajectories (e.g., raw videos), which requires structured guiderails during assessment. Our experimental results demonstrate LITEN is able to effectively learn from past experience to generate plans that use high-affordance instructions to accomplish long-horizon tasks.

Figures (9)

• Figure 1: Our method, Learning from Inference Time Execution (LITEN), is a novel approach that leverages a two-level hierarchical model to reason about complex tasks based on prior experience collected at inference-time, allowing it to learn the robot's affordances and gradually improve performance.
• Figure 2: An overview of our approach. LITEN cycles between (1) the reasoning phase, wherein the VLM reasons about the task to decompose the task into subtasks that the VLA low-level policy can roll out in sequence and (2) the assessment phase, wherein the VLM judges the outcomes of inference-time rollouts. The reasoning phase includes the outputs of the assessment phase by including them in the VLM's context, allowing the reasoner to iteratively learn robot affordances and improve performance.
• Figure 3: Examples of VLM judge's prompt and output for three subtasks (abridged for length). These generations are included in-context for the VLM reasoner in subsequent iterations attempting the task. Additional full examples are available in our codebase at https://github.com/ameesh-shah/liten-vla
• Figure 4: Success rates for the full completion of each multi-stage task over five attempt iterations. Our results show that LITEN is able to make effective use of prior attempts and consistently improves its plans as more experience is collected. Results are averaged over 10 trials.
• Figure 5: Examples of initial configurations and solutions for our tasks.