Affostruction: 3D Affordance Grounding with Generative Reconstruction

TL;DR

Affostruction tackles affordance grounding from RGBD by first reconstructing complete object geometry from partial observations using a multi-view sparse voxel fusion with constant token complexity and a flow-based generative prior. It then grounds open-vocabulary affordances on the reconstructed full shape via a flow-based model conditioned on CLIP text, capturing uncertainty with probabilistic heatmaps, and finally guides view selection by prioritizing viewpoints that reveal high-affordance regions. The approach extends TRELLIS with multi-view RGBD conditioning and an integrated affordance-flow module, achieving state-of-the-art results in both 3D reconstruction (IoU up to 32.67 on Toys4k) and complete-geometry affordance grounding (aIoU up to 19.1 on Affogato), while enabling efficient active-view sampling that accelerates functional understanding. This unified, uncertainty-aware framework improves robotic manipulation by enabling accurate reasoning about occluded functional regions and by optimizing data collection through affordance-driven view planning, with strong empirical gains over prior single-view or decoupled reconstruction-plus-grounding baselines.

Abstract

This paper addresses the problem of affordance grounding from RGBD images of an object, which aims to localize surface regions corresponding to a text query that describes an action on the object. While existing methods predict affordance regions only on visible surfaces, we propose Affostruction, a generative framework that reconstructs complete geometry from partial observations and grounds affordances on the full shape including unobserved regions. We make three core contributions: generative multi-view reconstruction via sparse voxel fusion that extrapolates unseen geometry while maintaining constant token complexity, flow-based affordance grounding that captures inherent ambiguity in affordance distributions, and affordance-driven active view selection that leverages predicted affordances for intelligent viewpoint sampling. Affostruction achieves 19.1 aIoU on affordance grounding (40.4\% improvement) and 32.67 IoU for 3D reconstruction (67.7\% improvement), enabling accurate affordance prediction on complete shapes.

Figures (9)

• Figure 1: Affostruction. Given an initial RGBD observation where functional regions for an affordance query (e.g., "attach a light fixture") are poorly visible, we perform generative reconstruction to complete the 3D geometry and predict affordances on the full shape including occluded surfaces using flow-based grounding. Our affordance-driven active view selection identifies optimal viewpoints (red) that maximize visibility of high-affordance regions while avoiding ineffective views (blue), prioritizing observation of functional regions during multi-view capture.
• Figure 2: Affostruction overview. Our approach consists of three stages. (1) Generative multi-view reconstruction: DINOv2 dinov2 features from multiple RGBD views are fused into sparse voxels using depth and camera parameters. A Flow Transformer conditioned on these multi-view features and trained with stochastic multi-view training extrapolates complete 3D structure from partial observations, decoded via frozen sparse structure decoder trellis (left). (2) Flow-based affordance grounding: A Sparse Flow Transformer conditioned on CLIP clip-encoded text query generates affordance heatmap logits over the reconstructed geometry (center). (3) Affordance-driven active view selection: We select next-best viewpoints by maximizing visibility of high-affordance regions, using frozen mesh decoder trellis for surface extraction (right). This enables affordance prediction on complete geometry from partial observations, with predicted affordances guiding view selection to prioritize functional regions.
• Figure 3: Diverse affordance predictions. Four sampling iterations for the same object-query pair produce diverse valid affordance distributions, demonstrating that our generative approach effectively captures the inherent ambiguity in affordance.
• Figure 4: Qualitative results on partial 3D affordance grounding. Affostruction reconstructs complete geometry and grounds affordances throughout entire objects from single RGBD views. Despite limited observations, our method predicts affordances on occluded regions, demonstrating the ability to reason about 3D functional interactions even when large portions of objects are unobserved.
• Figure 5: Multi-view training impact. We compare IoU (geometric reconstruction accuracy) as a function of the number of input views for methods trained with and without multi-view supervision. Methods trained on single views show minimal improvement or even degradation when given multiple views at inference (left group), while multi-view trained methods show consistent improvements with additional views (right group). Our method achieves the best performance in both settings, with multi-view training enabling effective exploitation of additional observations.