ProxyTransformation: Preshaping Point Cloud Manifold With Proxy Attention For 3D Visual Grounding

TL;DR

Proxy Transformation addresses the real-time ego-centric 3D visual grounding problem by refining the point cloud manifold using multimodal proxies. It introduces deformable point clustering to identify target submanifolds and a Proxy Attention mechanism to generate text-guided translations and image-guided transformations, yielding per-submanifold updates with linear complexity. The method achieves state-of-the-art performance on EmbodiedScan, with improvements of $7.49\%$ on easy targets and $4.60\%$ on hard targets, while cutting attention FLOPs by $40.6\%$, and it does not require offline scene-level point clouds. This work demonstrates the practical value of integrating language and vision proxies for real-time 3D grounding in embodied AI.

Abstract

Embodied intelligence requires agents to interact with 3D environments in real time based on language instructions. A foundational task in this domain is ego-centric 3D visual grounding. However, the point clouds rendered from RGB-D images retain a large amount of redundant background data and inherent noise, both of which can interfere with the manifold structure of the target regions. Existing point cloud enhancement methods often require a tedious process to improve the manifold, which is not suitable for real-time tasks. We propose Proxy Transformation suitable for multimodal task to efficiently improve the point cloud manifold. Our method first leverages Deformable Point Clustering to identify the point cloud sub-manifolds in target regions. Then, we propose a Proxy Attention module that utilizes multimodal proxies to guide point cloud transformation. Built upon Proxy Attention, we design a submanifold transformation generation module where textual information globally guides translation vectors for different submanifolds, optimizing relative spatial relationships of target regions. Simultaneously, image information guides linear transformations within each submanifold, refining the local point cloud manifold of target regions. Extensive experiments demonstrate that Proxy Transformation significantly outperforms all existing methods, achieving an impressive improvement of 7.49% on easy targets and 4.60% on hard targets, while reducing the computational overhead of attention blocks by 40.6%. These results establish a new SOTA in ego-centric 3D visual grounding, showcasing the effectiveness and robustness of our approach.

Figures (3)

• Figure 1: Illustration of our main idea and SOTA results. Ground truth and reference boxes are shown in green and purple respectively. Circular regions represent 3D areas where point cloud enhancement is applied. Black areas indicate regions that do not contribute to grounding performance and would increase unnecessary computation overhead. Traditional single-modality point cloud guidance would enhance these redundant areas. Red regions highlight areas where multimodal-guided point cloud enhancement is efficiently applied. Text modality, containing global relative position information among different critical objects, guides translation vectors for these region, while image modality, with local fine-grained semantic details, guides transformation matrices within each target regions. Our model achieves better results with reduced computation, about which details are in \ref{['table:val', 'tab:attn']}.
• Figure 2: (a) shows the overall framework of Proxy Transforamtion. For simplicity, $^\dagger$ indicates that a 2D grid is used to represent the 3D spatial grid, with generated 3D offsets also expressed as 2D vectors for clarity. The Grid Prior spans the entire space, and we illustrate it with only four reference points for clearer visualization. In Proxy Transformation module, $\mathcal{M}$ and $\mathcal{T}$ are sets of transformation matrixs and translation vectors for all clusters. (b) details the structure of our deformable offset network, and indicates the input and output shapes, where $M$ represents the number of clusters and $K$ represents the number of points per cluster. (c) illustrates the information flow in proxy attention. This module combines with FFN and skip connection in a standard Transformer architecture to form the Proxy Block.
• Figure 3: Visualization of ground truth and predictions. Ground truth boxes are shown in green, baseline in red, and ProxyTransformation's predictions in violet.