Efficient Prediction of Dense Visual Embeddings via Distillation and RGB-D Transformers

TL;DR

DVEFormer addresses the need for open-set, dense scene understanding in indoor robotics by learning pixel-wise, text-aligned embeddings through knowledge distillation from Alpha-CLIP. Built on an RGB‑D Transformer backbone, it distills teacher embeddings for each segment to train a lightweight student that outputs $768$-dimensional per-pixel embeddings, enabling both traditional closed-set segmentation via linear probing and flexible text-based querying. The method demonstrates competitive performance across NYUv2, SUN RGB-D, and ScanNet while delivering real-time inference on mobile hardware (26.3 FPS full model, 77.0 FPS with downsampling), and it integrates naturally into 3D mapping pipelines for robotics applications. These embeddings support retrieval by text prompts and can be leveraged to generate rich 3D scene representations, offering practical advantages for human-robot interaction and navigation in unstructured indoor environments.

Abstract

In domestic environments, robots require a comprehensive understanding of their surroundings to interact effectively and intuitively with untrained humans. In this paper, we propose DVEFormer - an efficient RGB-D Transformer-based approach that predicts dense text-aligned visual embeddings (DVE) via knowledge distillation. Instead of directly performing classical semantic segmentation with fixed predefined classes, our method uses teacher embeddings from Alpha-CLIP to guide our efficient student model DVEFormer in learning fine-grained pixel-wise embeddings. While this approach still enables classical semantic segmentation, e.g., via linear probing, it further enables flexible text-based querying and other applications, such as creating comprehensive 3D maps. Evaluations on common indoor datasets demonstrate that our approach achieves competitive performance while meeting real-time requirements, operating at 26.3 FPS for the full model and 77.0 FPS for a smaller variant on an NVIDIA Jetson AGX Orin. Additionally, we show qualitative results that highlight the effectiveness and possible use cases in real-world applications. Overall, our method serves as a drop-in replacement for traditional segmentation approaches while enabling flexible natural-language querying and seamless integration into 3D mapping pipelines for mobile robotics.

Figures (5)

• Figure 1: A real-world example of our proposed approach, called DVEFormer, predicting dense visual embeddings (DVE) on previously unseen data in one of our applications Cohumanics-Fischedick-ISR-2023. Our approach not only supports classical semantic segmentation (fixed predefined classes -- closed set) but also enables text-based queries to specify objects of interest beyond a fixed spectrum of predefined classes. Moreover, as shown on the bottom, the predicted embeddings can be easily integrated into existing mapping approaches, such as panopticndt2023iros. Semantic colors are chosen as in emsanet2022ijcnn and are https://github.com/TUI-NICR/nicr-scene-analysis-datasets/blob/v0.5.3/nicr_scene_analysis_datasets/datasets/nyuv2/nyuv2.py#L193NYUv2-eccv2012. We refer to Fig. \ref{['fig:t-sne']} for the subset of depicted semantic colors. Results for text-based queries show cosine similarity (red: high, white: low).
• Figure 2: Overview of our proposed approach on an example image of SUN RGB-D SUNRGBD-cvpr2015. Alpha‑CLIP is used offline to process RGB images and a set of binary segment masks to extract a set of teacher embeddings. These guide our efficient Dense Visual Embedding Transformer (DVEFormer) to learn dense pixel‑wise visual embeddings via knowledge distillation. For downstream applications, the resulting embeddings can be used for text‑based retrieval \ref{['fig:approach::application_querry']} or classical semantic segmentation with predefined classes \ref{['fig:approach::application_text_based']}-\ref{['fig:approach::application_linear_probing']}, or other robotic applications, enabling flexible scene understanding tasks. We refer to Fig. \ref{['fig:t-sne']} for the subset of depicted semantic colors.
• Figure 3: Visualization of segment embeddings generated by Alpha‑CLIP (see Sec. \ref{['sec:embedding_vector_estimation']}) for the NYUv2 training split: \ref{['fig:t-sne::sem']} shows the embeddings before removing global scene context, while \ref{['fig:t-sne::sem_diff']} depicts them after context suppression ($\alpha=0.65$). In both figures, in the main view, points are colored by the ground-truth semantic class, and the subplot shows the same points colored by their scene class (as defined in emsanet2022ijcnn). Note that in \ref{['fig:t-sne::sem']} clusters form primarily by scene class, whereas in \ref{['fig:t-sne::sem_diff']}, due to scene context suppression, clusters are more distinctly organized by semantic class. The embeddings were reduced using PCA and t‑SNE using cosine similarity as the distance metric.
• Figure 4: Closed-set results for classical semantic segmentation on the NYUv2 test split for different configurations (see all legends and the x-axis). Note that ground-truth masks are used for computing the GT performance and, thus, assume perfect segmentation. Therefore, the gap between the learned models and the GT is not solely due to the knowledge distillation process. Throughput was measured on an NVIDIA Jetson AGX Orin (JetPack 6.2, TensorRT 10.3, Float16). Additional details on the metrics and experimental setup are provided in Sec. \ref{['sec:nyuv2_results']}.
• Figure 5: 3D-NDT mapping results for a ScanNet validation scene (valid scene0608). The map shows classical semantic segmentation (via linear probing) as well as text-based queries (cosine similarity: red: high, white: low). Additionally, an RGB image for a single pose is shown for the same text queries. We refer to Fig. \ref{['fig:t-sne']} for the subset of depicted semantic colors.