LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension

TL;DR

This paper tackles the gap between zero-shot Vision-Language Models (VLMs) and task-tuned models in Referring Expression Comprehension (REC) by proposing LLM-wrapper, a black-box adaptation framework. It leverages Large Language Models (LLMs) to reason over VLM outputs that are translated into natural language prompts, and fine-tunes the LLM with LoRA to predict the best bounding box among candidates, without requiring access to the VLM’s internals. Across REC benchmarks (RefCOCO, RefCOCO+, RefCOCOg, Talk2Car) and HC-RefLoCo, and for multiple VLMs (Grounding-DINO and Florence-2) and LLMs (Mixtral and Llama 3 8B), LLM-wrapper yields substantial improvements in $P@1$ over zero-shot baselines, demonstrates transfer to new datasets, and enables model ensembling. The approach is lightweight, model-agnostic, and compatible with existing REC-tuned VLMs, offering a practical route for adapting proprietary or API-based models while preserving their original capabilities.

Abstract

Vision Language Models (VLMs) have demonstrated remarkable capabilities in various open-vocabulary tasks, yet their zero-shot performance lags behind task-specific fine-tuned models, particularly in complex tasks like Referring Expression Comprehension (REC). Fine-tuning usually requires 'white-box' access to the model's architecture and weights, which is not always feasible due to proprietary or privacy concerns. In this work, we propose LLM-wrapper, a method for 'black-box' adaptation of VLMs for the REC task using Large Language Models (LLMs). LLM-wrapper capitalizes on the reasoning abilities of LLMs, improved with a light fine-tuning, to select the most relevant bounding box matching the referring expression, from candidates generated by a zero-shot black-box VLM. Our approach offers several advantages: it enables the adaptation of closed-source models without needing access to their internal workings, it is versatile as it works with any VLM, it transfers to new VLMs and datasets, and it allows for the adaptation of an ensemble of VLMs. We evaluate LLM-wrapper on multiple datasets using different VLMs and LLMs, demonstrating significant performance improvements and highlighting the versatility of our method. While LLM-wrapper is not meant to directly compete with standard white-box fine-tuning, it offers a practical and effective alternative for black-box VLM adaptation. Code and checkpoints are available at https://github.com/valeoai/LLM_wrapper .

Figures (10)

• Figure 1: Illustration of LLM-wrapper. Our method adapts a black-box VLM for the REC task. The VLM output is translated into natural language to prompt an LLM. The latter is tasked with identifying the box that best matches the query among the given candidates. The LLM must learn to identify the query's subject and to disambiguate the correct object from distractors (e.g., several plates with 'flower details').
• Figure 2: Qualitative results of Flo2 on RefCOCOg (first row) and Talk2Car (second row), before and after adaptation with LLM-wrapper, provided with queries as captions. Adapting Flo2 (in orange) with LLM-wrapper (using Llama 3 8B, in blue) leads to improved reasoning and box selection.
• Figure 3: Visualizations of the candidates and predictions for the query "A bottle of wine between the vegetables". We visualize the ground truth (a) and the set of box candidates generated by Flo2 (b) and GDrec (c). In the second row, we visualize the final predictions of Flo2 (e) and GDrec (f) and in (d) the prediction of LLM-wrapper applied on the ensemble of both VLMs' outputs, using Llama 3 8B. We observe that LLM-wrapper discards the distractor bottle and selects the correct object.
• Figure 4: Study of LLM-wrapper's sensitivity to hyper-parameters. Impact on LLM-wrapper's performances (with GD on RefCOCOg) of the (a) LLM scale; (b) LoRA rank $r$; and (c) learning rate.
• Figure 5: Performance (P@1) of LLM-wrapper on RefCOCOg (val) with respect to the number of training samples. We fine-tune Llama 3 8B on RefCOCOg (train).