Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking

TL;DR

The paper addresses grounding open-vocabulary perception in dynamic robotic environments by integrating vision-language models with open-vocabulary detectors and fast video segmentation. It introduces VLM-GIST, a pipeline that converts VLM-derived structured descriptions into prompts for grounding, followed by segmentation and online tracking, with a low-frequency update mechanism to maintain efficiency. Key contributions include an instance-aware grounding assignment method, a validation-based error-correction step, and a comprehensive evaluation on real robots and benchmarks, plus public-release of code, data, and datasets. The approach enables task-specific attribute extraction and automatic dataset annotation, supporting robust, scalable perception for robot manipulation and interaction in open-world settings.

Abstract

Vision-language models (VLMs) excel in visual understanding but often lack reliable grounding capabilities and actionable inference rates. Integrating them with open-vocabulary object detection (OVD), instance segmentation, and tracking leverages their strengths while mitigating these drawbacks. We utilize VLM-generated structured descriptions to identify visible object instances, collect application-relevant attributes, and inform an open-vocabulary detector to extract corresponding bounding boxes that are passed to a video segmentation model providing segmentation masks and tracking. Once initialized, this model directly extracts segmentation masks, processing image streams in real time with minimal computational overhead. Tracks can be updated online as needed by generating new structured descriptions and detections. This combines the descriptive power of VLMs with the grounding capability of OVD and the pixel-level understanding and speed of video segmentation. Our evaluation across datasets and robotics platforms demonstrates the broad applicability of this approach, showcasing its ability to extract task-specific attributes from non-standard objects in dynamic environments. Code, data, videos, and benchmarks are available at https://vlm-gist.github.io

Figures (6)

• Figure 1: Visualization of detected object instances, including object classes, visual descriptions, bounding boxes, and segmentation masks, using off-the-shelf foundation models without prior knowledge of the image content. Note, that some object descriptions are collapsed to reduce visual clutter.
• Figure 2: Pipeline leveraging vision-language models for open-vocabulary instance segmentation and tracking. The low-frequency update mechanism initializes the tracker and updates tracks on demand. The lightweight tracker generates high-frequency segmentation masks on the full image stream.
• Figure 3: Object instances with four additional user-defined attributes parsed from a structured description annotating their corresponding detections.
• Figure 4: Experimental robot platforms. (a) Team NimbRo's TIAGo++ robot at the RoboCup@Home 2024 finals in Eindhoven, grasping ingredients associated with a dinner recipe. (b) Industrial scenario where objects on a conveyor belt are detected from above, picked up, and placed in boxes.
• Figure 5: Manipulation scenes and corresponding tasks which were successfully solved by a mobile manipulator equipped with our proposed method.