3DAffordSplat: Efficient Affordance Reasoning with 3D Gaussians

TL;DR

This work introduces 3DAffordSplat, the first large-scale, multi-modal dataset for 3D Gaussian Splatting (3DGS) based affordance reasoning, paired with AffordSplatNet, a cross-modal model that aligns dense Gaussian representations with sparse point clouds and natural language. The approach leverages Gaussian-text fusion, granularity-adaptive decoding, and a Cross-Modal Structure Alignment (CMSA) pretraining objective to bridge modalities and improve robustness to geometry, occlusion, and unseen objects. Empirical results show significant gains over point-cloud baselines, with strong Seen and improved Unseen generalization, underscoring the value of dense 3DGS representations for affordance understanding. The dataset and model enable robust, language-guided affordance reasoning, with broad potential applications in robotic manipulation, AR interfaces, and smart environments where accurate spatial interaction cues are critical.

Abstract

3D affordance reasoning is essential in associating human instructions with the functional regions of 3D objects, facilitating precise, task-oriented manipulations in embodied AI. However, current methods, which predominantly depend on sparse 3D point clouds, exhibit limited generalizability and robustness due to their sensitivity to coordinate variations and the inherent sparsity of the data. By contrast, 3D Gaussian Splatting (3DGS) delivers high-fidelity, real-time rendering with minimal computational overhead by representing scenes as dense, continuous distributions. This positions 3DGS as a highly effective approach for capturing fine-grained affordance details and improving recognition accuracy. Nevertheless, its full potential remains largely untapped due to the absence of large-scale, 3DGS-specific affordance datasets. To overcome these limitations, we present 3DAffordSplat, the first large-scale, multi-modal dataset tailored for 3DGS-based affordance reasoning. This dataset includes 23,677 Gaussian instances, 8,354 point cloud instances, and 6,631 manually annotated affordance labels, encompassing 21 object categories and 18 affordance types. Building upon this dataset, we introduce AffordSplatNet, a novel model specifically designed for affordance reasoning using 3DGS representations. AffordSplatNet features an innovative cross-modal structure alignment module that exploits structural consistency priors to align 3D point cloud and 3DGS representations, resulting in enhanced affordance recognition accuracy. Extensive experiments demonstrate that the 3DAffordSplat dataset significantly advances affordance learning within the 3DGS domain, while AffordSplatNet consistently outperforms existing methods across both seen and unseen settings, highlighting its robust generalization capabilities.

Figures (12)

• Figure 1: Dataset overview. (a) Category distribution in 3DAffordSplat. (b) Numbers of 3DGS annotations in each affordance category. (c) Representative data examples from 3DAffordSplat (3DGS and point cloud, with affordance annotations and questions), the colored region in point clouds and 3DGS is the affordance annotation. (d) Examples of affordance reasoning.
• Figure 2: Compared to sparse point clouds, 3DGS provides more vivid textures and clearer geometry. 3DGS-based Affordances can capture more complex structures. Moreover, the continuous nature of Gaussians supports smooth affordance representation over surfaces and even curves.
• Figure 3: Architecture Overview.AffordSplatNet (a) processes 3D Gaussians and human instructions through a hierarchical pipeline. It extracts multi-granularity features from Gaussians, while a pre-trained language model infers an $\langle \text{Aff} \rangle$ token from the text query, representing an intermediate segmentation result. These modalities are fused through attention mechanisms, with granularity selection prioritizing task-relevant spatial scales. The selected features decode into dynamic kernels for efficient affordance mask generation. To enhance 3D structural learning, Cross-Modal Structure Alignment (CMSA) (b) module aligns the Affordance regions and overall structural relations between the Gaussian and point cloud data at the structural level.
• Figure 4: Visualization Results of AffordSplatNet. Each example includes one query, one answer and four object shapes, illustrating the model's generalization capability in affordance knowledge. The identified affordance regions are marked in red.
• Figure 5: Real-world cases. Two common objects are shown.