TRACE: Textual Reasoning for Affordance Coordinate Extraction

TL;DR

TRACE introduces a textual Chain of Reasoning (CoR) to bridge instruction semantics and precise affordance coordinates in robotic grounding. The approach couples a Vision-Language Model with a large language model through a multimodal projection and a reasoning-augmented TRACE dataset (200k samples: 100k CoR-augmented plus 100k standard instruction-tuning), enabling the model to externalize its spatial reasoning before acting. Through instruction-tuning on TRACE, the model achieves state-of-the-art results on Where2Place benchmarks (48.1% on W2P and 55.0% on W2P(h)) and exhibits a dose-dependent improvement with more reasoning data, alongside interpretable attention dynamics. The work provides dataset and code release, and suggests that textual CoR is a robust pathway to more precise, reliable, and explainable VLM-driven robot control.

Abstract

Vision-Language Models (VLMs) struggle to translate high-level instructions into the precise spatial affordances required for robotic manipulation. While visual Chain-of-Thought (CoT) methods exist, they are often computationally intensive. In this work, we introduce TRACE (Textual Reasoning for Affordance Coordinate Extraction), a novel methodology that integrates a textual Chain of Reasoning (CoR) into the affordance prediction process. We use this methodology to create the TRACE dataset, a large-scale collection created via an autonomous pipeline that pairs instructions with explicit textual rationales. By fine-tuning a VLM on this data, our model learns to externalize its spatial reasoning before acting. Our experiments show that our TRACE-tuned model achieves state-of-the-art performance, reaching 48.1% accuracy on the primary Where2Place (W2P) benchmark (a 9.6% relative improvement) and 55.0% on the more challenging W2P(h) subset. Crucially, an ablation study demonstrates that performance scales directly with the amount of reasoning data used, confirming the CoR's effectiveness. Furthermore, analysis of the model's attention maps reveals an interpretable reasoning process where focus shifts dynamically across reasoning steps. This work shows that training VLMs to generate a textual CoR is an effective and robust strategy for enhancing the precision, reliability, and interpretability of VLM-based robot control. Our dataset and code are available at https://github.com/jink-ucla/TRACE

Figures (9)

• Figure 1: An example data point from our TRACE reasoning dataset illustrating its overall structure. Each entry consists of an image and a corresponding natural language question that requires spatial reasoning, such as finding the vacant area between the two delineated objects. The dataset also provides the explicit, multi-step reasoning process required to solve the instruction. This process includes identifying reference objects, determining the goal's subtype as a "Placement Affordance", defining the search space on the appliance's surface, and generating the final coordinates as a list of tuples.
• Figure 2: Overview of the model's reasoning pipeline. Given an image and a corresponding natural language instruction, the system begins a multi-step reasoning process. The model first determines the subtype of the goal and establishes the relevant reference surface based on the image. It then defines the target area by interpreting the request. This process utilizes an Image Encoder, Tokenizer, and a Large Language Model (LLM). Finally, a Projection MLP generates the output, which consists of normalized coordinates for points within the identified vacant space.
• Figure 3: A qualitative comparison of TRACE with other leading models on a sample from our reasoning dataset. The given instruction is to "Pinpoint several spots in the vacant area that lies to the right of the glass container".
• Figure 4: Ablation study on the impact of reasoning data. The plot shows the performance on the RoboRefIt, Where2Place (W2P), and Where2Place (h) benchmarks as the percentage of the TRACE reasoning dataset used for training is increased from 0% (baseline) to 100%. The solid lines represent the trend (linear regression), while the shaded areas indicate the 95% confidence interval. Performance across all tasks consistently improves with more reasoning data, providing strong evidence for the effectiveness of our approach.
• Figure 5: A visualization of the model's reasoning attention map for the instruction: "Find the free space in front of the window on the left." The figure illustrates the model's focus across the four-step textual reasoning process. The attention heatmap is overlaid on the input image. (1) Identify Reference Object and (2) Define Target Area: The model exhibits diffuse, weak attention during the initial steps of identifying the window and defining the target area. (3) Determine Goal's Subtype: Attention begins to focus as the model classifies the task. A distinct high-attention region (bright orange) emerges over the target area, suggesting this is a critical reasoning step. (4) Generate Output: In the final step, there is minimal attention on the image, especially over the predicted points (shown as dots), indicating the model relies on its completed textual reasoning to generate the final coordinates rather than direct, concurrent visual evidence.