TRACER: Texture-Robust Affordance Chain-of-Thought for Deformable-Object Refinement

TL;DR

TRACER tackles the challenge of grounding deformable-object affordances under varied textures by bridging high-level semantic reasoning and low-level perception. It introduces TA-CoT for hierarchical task decomposition, SCBR to constrain spatial boundaries, and ICRF for convergent refinement of scattered predictions, enabling robust long-horizon manipulation. Experiments on Fine-AGDDO15 and a real-world dual-arm platform show substantial gains in grounding precision and task success, demonstrating effective translation from semantic plans to physically executable interaction regions. The work advances practical, texture-robust affordance grounding and provides open-source resources to foster further research in semantic-to-perception integration for deformable-object robotics.

Abstract

The central challenge in robotic manipulation of deformable objects lies in aligning high-level semantic instructions with physical interaction points under complex appearance and texture variations. Due to near-infinite degrees of freedom, complex dynamics, and heterogeneous patterns, existing vision-based affordance prediction methods often suffer from boundary overflow and fragmented functional regions. To address these issues, we propose TRACER, a Texture-Robust Affordance Chain-of-thought with dEformable-object Refinement framework, which establishes a cross-hierarchical mapping from hierarchical semantic reasoning to appearance-robust and physically consistent functional region refinement. Specifically, a Tree-structured Affordance Chain-of-Thought (TA-CoT) is formulated to decompose high-level task intentions into hierarchical sub-task semantics, providing consistent guidance across various execution stages. To ensure spatial integrity, a Spatial-Constrained Boundary Refinement (SCBR) mechanism is introduced to suppress prediction spillover, guiding the perceptual response to converge toward authentic interaction manifolds. Furthermore, an Interactive Convergence Refinement Flow (ICRF) is developed to aggregate discrete pixels corrupted by appearance noise, significantly enhancing the spatial continuity and physical plausibility of the identified functional regions. Extensive experiments conducted on the Fine-AGDDO15 dataset and a real-world robotic platform demonstrate that TRACER significantly improves affordance grounding precision across diverse textures and patterns inherent to deformable objects. More importantly, it enhances the success rate of long-horizon tasks, effectively bridging the gap between high-level semantic reasoning and low-level physical execution. The source code and dataset will be made publicly available at https://github.com/Dikay1/TRACER.

Figures (11)

• Figure 1: Affordance grounding under different semantic conditions. (a) Vision-only: Prediction based only on visual input. (b) Instruction-conditioned: Guided by a single-step language instruction. (c) CoT-conditioned (Ours): Conditioned on hierarchical sub-task semantics from TA-CoT. Ego: first-person view; Exo: third-person view.
• Figure 2: Overview of the proposed TRACER framework. High-level semantic instructions are hierarchically decomposed via Tree-structured Affordance Chain-of-Thought (TA-CoT), followed by spatial refinement through Spatially-Constrained Boundary Refinement (SCBR) and Interactive Convergence Refinement Flow (ICRF) to achieve physically consistent affordance grounding. The resulting grounding maps then guide closed-loop execution on the bimanual robotic platform.
• Figure 3: Overview of the Fine-AGDDO15 dataset construction pipeline, together with the two-stage training and evaluation strategy.
• Figure 4: Four-state gating mechanism for TA-CoT reasoning. Hierarchical paths are dynamically managed via accept, reject, dormant, and feedback states.
• Figure 5: Comparative visualization of ICRF flow field dynamics. (a) Raw gradient field without flow matching. (b) Convergence refinement flow field generated by ICRF. The $\textcolor{orange}{\star}$ denotes the initial state $x_0$, and the $\textcolor{green!60!black}{\star}$ denotes the target manipulation point $x_1$.