Configurable Embodied Data Generation for Class-Agnostic RGB-D Video Segmentation

TL;DR

This letter presents a method for generating large-scale datasets to improve class-agnostic video segmentation across robots with different form factors and demonstrates that using MVPd for finetuning can lead to performance improvements when transferring foundation models to certain robot embodiments, such as specific camera placements.

Abstract

This paper presents a method for generating large-scale datasets to improve class-agnostic video segmentation across robots with different form factors. Specifically, we consider the question of whether video segmentation models trained on generic segmentation data could be more effective for particular robot platforms if robot embodiment is factored into the data generation process. To answer this question, a pipeline is formulated for using 3D reconstructions (e.g. from HM3DSem) to generate segmented videos that are configurable based on a robot's embodiment (e.g. sensor type, sensor placement, and illumination source). A resulting massive RGB-D video panoptic segmentation dataset (MVPd) is introduced for extensive benchmarking with foundation and video segmentation models, as well as to support embodiment-focused research in video segmentation. Our experimental findings demonstrate that using MVPd for finetuning can lead to performance improvements when transferring foundation models to certain robot embodiments, such as specific camera placements. These experiments also show that using 3D modalities (depth images and camera pose) can lead to improvements in video segmentation accuracy and consistency. The project webpage is available at https://topipari.com/projects/MVPd

Figures (6)

• Figure 1: Illustration of the data generation pipeline used to create MVPd. Left: Using input meshes that contain RGB and segmentation textures, the pipeline generates sparse random paths with a collision-free NavMesh planner habitat19iccv and interpolates them into dense trajectories of way poses. Right: The motion trajectories are refined according to an embodiment configuration file and rendered to output videos.
• Figure 2: Impact of specific embodiments (sensor placement & active illumination) on visual features rendered by the MVPd
• Figure 3: Distribution of the 20 object clusters used for creating the zero-shot subset of MVPd. Object instances from the 'Objects' and 'Misc' super-categories in MVPd, as defined in Matterport3D, are grouped into 20 clusters using $k$-means with the CLIP-embedding radford2021clip of each instance's human-annotated text description. Clusters are then ranked by the number of associated training videos, and the smallest 20% (clothes storage, hobby items, stands, and soap) are defined as the zero-shot classes. The text summary of each cluster (e.g. Clothes Storage, Hobby, etc.) are defined based on manual inspection.
• Figure 4: Illustration of the the class-agnostic video segmentation quality (VSQ) metric. Given a fixed $k$-frame window size, ground truth and predicted segments are isolated into segment tubes and then matched using an optimal assignment algorithm according to pairwise F-measure. Once matched, the set of true positive, false positive and false negative tubes are counted and a per-match $IoU$ is computed.
• Figure 5: Qualitative results comparing the swin-variants of Video K-Net, Tube-Link, OV2Seg, and FastSPAM. FastSPAM exhibits segments that are more consistent and accurate than the baselines and with fewer false positive predictions (outlined in neon yellow).