VLC Fusion: Vision-Language Conditioned Sensor Fusion for Robust Object Detection

TL;DR

VLC Fusion addresses the instability of multi-modal object detection under diverse environmental conditions by conditioning sensor fusion on environmental cues derived from Vision-Language Models. It introduces an offline condition extraction pipeline and a CBAM+FiLM fusion module that dynamically weights modalities via a condition vector $r_x$. Empirical results on Waymo and ATR show consistent improvements in seen and unseen scenarios, with ablations showing the importance of semantically consistent conditions and the trade-off between condition richness and noise. This approach enables more robust autonomous perception and can be extended to additional modalities and real-time deployment.

Abstract

Although fusing multiple sensor modalities can enhance object detection performance, existing fusion approaches often overlook subtle variations in environmental conditions and sensor inputs. As a result, they struggle to adaptively weight each modality under such variations. To address this challenge, we introduce Vision-Language Conditioned Fusion (VLC Fusion), a novel fusion framework that leverages a Vision-Language Model (VLM) to condition the fusion process on nuanced environmental cues. By capturing high-level environmental context such as as darkness, rain, and camera blurring, the VLM guides the model to dynamically adjust modality weights based on the current scene. We evaluate VLC Fusion on real-world autonomous driving and military target detection datasets that include image, LIDAR, and mid-wave infrared modalities. Our experiments show that VLC Fusion consistently outperforms conventional fusion baselines, achieving improved detection accuracy in both seen and unseen scenarios.

Figures (19)

• Figure 1: Overview of VLC Fusion. Compared to (a) standard fusion for object detection, (b) our method modulates modality-specific features with environment-specific meta-information, called conditions, improving the resilience of object detection to diverse natural environmental variations.
• Figure 2: Comparision of sample predictions from multi-modal fusion and VLC Fusion: from left to right, raw LiDAR point clouds and camera views, predictions from a multi‐modal fusion baseline, and predictions from our VLC Fusion conditioned on VLM‐extracted environmental cues (✓/✗ prompts shown at right). Conditioning on high‐level context improves detection performance in recovering occluded cars (Example 1), detecting more vehicles under nighttime glare (Example 2), and correctly identifying the cyclist (magenta) and pedestrians (orange) (Example 3) where the baseline misclassifies.
• Figure 3: Overview of the three-step automated pipeline for extracting environmental conditions.
• Figure 4: Qualitative examples of VLC Fusion in both seen and unseen environments. Top row: 3D detections on the Waymo Open Dataset under seen (Day and Night time) and unseen (Dawn/dusk time) conditions, with vehicles (yellow), cyclists (purple) and pedestrians (red) accurately localized. Bottom row: 2D detection on the ATR dataset for seen (1000 m and 2000 m distances respectively) and unseen (1500 m distance) scenarios. More qualitative examples are provided in Appendix \ref{['app:qualitative']}.
• Figure 5: (a) Model parameter counts and per-image inference times highlight the efficiency gains of Moondream2 and SmolVLM-Instruct over the GPT-4o baseline. (b) In zero-shot 3D mAP tests on Day–Night and Dawn/Dusk scenarios of the Waymo Open Dataset, both small-scale models achieve scores comparable to GPT-4o.