Scaling Test-time Inference for Visual Grounding

TL;DR

EGM tackles the latency-accuracy gap in visual grounding for small VLMs by scaling test-time token generation. It introduces a two-stage SFT-RL training pipeline that endows small models with multi-modal reasoning, leveraging GPT-generated reasoning traces for vanilla and amodal grounding and learnability-based RL data curation with a token-level GRPO objective. The reward combines $IoU$ and grounding success, guiding the model to reason through complex prompts and occlusions. Across InternVL and QwenVL families, EGM yields consistent improvements in vanilla and amodal grounding and achieves a substantial latency reduction, e.g., an 8B model attaining $IoU=91.4$ on RefCOCO while being several times faster than much larger baselines, highlighting practical deployment benefits.

Abstract

Visual grounding is an essential capability of Visual Language Models (VLMs) to understand the real physical world. Previous state-of-the-art grounding visual language models usually have large model sizes, making them heavy for deployment and slow for inference. However, we notice that the sizes of visual encoders are nearly the same for small and large VLMs and the major difference is the sizes of the language models. Small VLMs fall behind larger VLMs in grounding because of the difference in language understanding capability rather than visual information handling. To mitigate the gap, we introduce 'Efficient visual Grounding language Models' (EGM): a method to scale the test-time computation (#generated tokens). Scaling the test-time computation of a small model is deployment-friendly, and yields better end-to-end latency as the cost of each token is much cheaper compared to directly running a large model. On the RefCOCO benchmark, our EGM-Qwen3-VL-8B demonstrates 91.4 IoU with an average of 737ms (5.9x faster) latency while Qwen3-VL-235B demands 4,320ms to achieve 90.5 IoU. To validate our approach's generality, we further set up a new amodal grounding setting that requires the model to predict both the visible and occluded parts of the objects. Experiments show our method can consistently and significantly improve the vanilla grounding and amodal grounding capabilities of small models to be on par with or outperform the larger models, thereby improving the efficiency for visual grounding.

Figures (12)

• Figure 1: Overview of test-time scaling for visual grounding.Left. Existing state-of-the-art grounding VLMs usually have large model sizes. Right. Our EGM extends scaling laws by scaling up inference tokens to enhance text understanding capabilities of small VLMs, achieving better efficiency.
• Figure 2: Failure cases of small VLMs. We find small VLMs, e.g., InternVL-3-8B, tend to fail when the text prompt is semantically complicated and there are multiple candidates in the image that can confuse the model. We term this failure pattern 'COMPLEX_PROMPT' and label the ground truth bbox in blue, and the 8B model prediction in orange in examples.
• Figure 3: Overview of our method.Top (a): Data curation pipeline of training data with reasoning. We feed the image, text prompt and ground truth bounding box of the target object into a proprietary VLM to generate the detailed reasoning process of how to locate the object correctly given the image and text prompt. The generated reasoning process is incorporated as part of the training data. Bottom (b): Examples of generated reasoning training data for vanilla grounding and amodal grounding. The reasoning process of vanilla grounding analyzes the feature that distinguishes the target object from others, and the reasoning process of amodal grounding further involves what object causes the occlusion and in which directions the visible parts should be extended to recover the complete object.
• Figure 4: Accuracy vs. Efficiency. Our models, EGM-Qwen3-VL-2B, EGM-Qwen3-VL-4B and EGM-Qwen3-VL-8B, have greatly improved the efficiency of visual grounding. For example, EGM-Qwen3-VL-8B outperforms both the state-of-the-art Qwen3-VL-235B-Instruct and Qwen3-VL-235B-Thinking models for accuracy, while speeding up 5.9$\times$/18.9$\times$ in terms of GPU latency. For Qwen models, '-T' denotes '-Thinking' and '-I' denotes '-Instruct'.
• Figure 5: Prompt for training and inference of models from two different families.