QueryAdapter: Rapid Adaptation of Vision-Language Models in Response to Natural Language Queries

TL;DR

QueryAdapter addresses the problem of adapting vision-language models to robotic environments under open-vocabulary natural language queries without predefined closed sets. It uses a large language model to generate target classes for each query, leverages unlabelled deployment data to align VLM features with those targets, and tunes a small set of prompt vectors while selecting relevant training objects via top-k sampling. Negative class labels derived from object captions calibrate confidence and help handle open-query objects. Experiments on ScanNet++ show significant improvements over unsupervised VLM adapters and 3D scene graph baselines, with robust generalization to affordance queries and Ego4D, demonstrating practical, minutes-scale adaptation for real-world robotics.

Abstract

A domain shift exists between the large-scale, internet data used to train a Vision-Language Model (VLM) and the raw image streams collected by a robot. Existing adaptation strategies require the definition of a closed-set of classes, which is impractical for a robot that must respond to diverse natural language queries. In response, we present QueryAdapter; a novel framework for rapidly adapting a pre-trained VLM in response to a natural language query. QueryAdapter leverages unlabelled data collected during previous deployments to align VLM features with semantic classes related to the query. By optimising learnable prompt tokens and actively selecting objects for training, an adapted model can be produced in a matter of minutes. We also explore how objects unrelated to the query should be dealt with when using real-world data for adaptation. In turn, we propose the use of object captions as negative class labels, helping to produce better calibrated confidence scores during adaptation. Extensive experiments on ScanNet++ demonstrate that QueryAdapter significantly enhances object retrieval performance compared to state-of-the-art unsupervised VLM adapters and 3D scene graph methods. Furthermore, the approach exhibits robust generalization to abstract affordance queries and other datasets, such as Ego4D.

Figures (5)

• Figure 1: Existing methods for overcoming the domain gap between captioned images and robotic data streams require the definition of a closed-set of classes. This is unrealistic for robots that detect objects in response to diverse natural language queries. In response, we explore how a pre-trained VLM could be rapidly adapted to natural language queries as they arise. This approach avoids having to pre-define a closed-set of classes, ensuring that an adapted model can be used for open-vocabulary object detection.
• Figure 2: Our proposed framework for rapidly adapting a pre-trained VLM to detect objects relevant to a natural language query. Given a new query, we use an LLM to generate a set of "target classes" required to fulfill the request. Unlabelled data collected by the robot in previous deployments is then used to align VLM features with these target classes. As a final step, the adapted model is used to detect the target classes in the current scene.
• Figure 3: A detailed summary of QueryAdapter, the proposed framework for responding to natural language queries with an adapted VLM. The method is split into four steps; object captioning and storage, training data selection, adaptation and object retrieval.
• Figure 4: Comparison of QueryAdapter with state-of-the-art unsupervised VLM adapters and 3DSG methods.
• Figure 5: Impact of key parameters on QueryAdapter performance and training time using the small sets of target classes. The purple solid lines show the performance of the adapted model on the target classes. The purple dotted lines refers to performance of the pre-trained system on the target classes. The orange solid line shows the time taken to perform adaptation.