RGBT-Ground Benchmark: Visual Grounding Beyond RGB in Complex Real-World Scenarios

TL;DR

This work tackles the limitations of RGB-only visual grounding benchmarks by introducing RGBT-Ground, the first large-scale RGB–Thermal VG benchmark with multi-level scene, environment, and object annotations to reflect real-world variability. It establishes a unified RGBT-VG framework for fair cross-modal evaluation and proposes RGBT-VGNet, a CLIP-based baseline that uses Asymmetric Modality Adaptation and Language-Aware Visual Synergy to fuse RGB and TIR cues under referring expressions. Extensive experiments across multiple sub-datasets and settings reveal that multi-modal fusion markedly improves robustness, especially in night-time and distant scenarios, highlighting the value of RGB–TIR information for reliable grounding. The authors also release all data, framework code, and evaluation tools to promote progress toward robust, real-world vision–language grounding systems.

Abstract

Visual Grounding (VG) aims to localize specific objects in an image according to natural language expressions, serving as a fundamental task in vision-language understanding. However, existing VG benchmarks are mostly derived from datasets collected under clean environments, such as COCO, where scene diversity is limited. Consequently, they fail to reflect the complexity of real-world conditions, such as changes in illumination, weather, etc., that are critical to evaluating model robustness and generalization in safety-critical applications. To address these limitations, we present RGBT-Ground, the first large-scale visual grounding benchmark built for complex real-world scenarios. It consists of spatially aligned RGB and Thermal infrared (TIR) image pairs with high-quality referring expressions, corresponding object bounding boxes, and fine-grained annotations at the scene, environment, and object levels. This benchmark enables comprehensive evaluation and facilitates the study of robust grounding under diverse and challenging conditions. Furthermore, we establish a unified visual grounding framework that supports both uni-modal (RGB or TIR) and multi-modal (RGB-TIR) visual inputs. Based on it, we propose RGBT-VGNet, a simple yet effective baseline for fusing complementary visual modalities to achieve robust grounding. We conduct extensive adaptations to the existing methods on RGBT-Ground. Experimental results show that our proposed RGBT-VGNet significantly outperforms these adapted methods, particularly in nighttime and long-distance scenarios. All resources will be publicly released to promote future research on robust visual grounding in complex real-world environments.

Figures (9)

• Figure 1: Comparative overview of the RGBT-Ground benchmark and its evaluation of representative grounding method. (a) Comparative characteristics of RGBT-Ground against classical visual grounding benchmarks. (b) Data distribution of lighting, occlusion, object, weather, and scene types. (c) Method evaluation of the representative grounding method and our RGBT-VGNet baseline on RGBT-Ground.
• Figure 2: Overview and characteristic analysis of the proposed RGBT-Ground benchmark. (a–c) present dataset distributions and correlations across scene, environment, and object level annotation, while (d–e) show object location distributions and representative samples. Compared with prior visual grounding datasets, RGBT-Ground captures more realistic conditions with off-center, small, and occluded objects under low-light and adverse weather, supported by paired RGB–Thermal imagery.
• Figure 3: Illustration of the RGBT-Ground data preparation pipeline. The process consists of three main stages: (1) Collection of multi-modal RGBT source data; (2) filtering based on the predefined rules, and (3) captioning using carefully designed prompts with the Qwen-VL qwen-vl model.
• Figure 4: The architecture of the proposed multi-modal RGBT Visual Grounding baseline method (RGBT-VGNet).
• Figure 5: Qualitative comparison of multi-modal RGBT visual grounding results on RGBT-Ground. The three rows correspond to RefM$^3$FD, RefMFAD, and RefFLIR sub-datasets, respectively. Bounding box colors match the description below. Each column shows a different visual grounding method, stressing differences across methods under diverse illumination, weather, object size, and occlusion.