SLGNet: Synergizing Structural Priors and Language-Guided Modulation for Multimodal Object Detection

TL;DR

This paper addresses robust multimodal object detection under challenging RGB-IR conditions by preserving cross-modal structural cues and enabling environmental awareness through language-guided modulation. It introduces SLGNet, a parameter-efficient framework that freezes a Vision Transformer backbone and adds two modules: a Structure-Aware Adapter (SA-Adapter) to recover geometric details via a Structure Encoder and a Hierarchical Sparse Attention-based FF-Adapter, and a Language-Guided Modulation (LGM) that uses VLM-derived structured captions to dynamically recalibrate features. The approach achieves state-of-the-art results across LLVIP, FLIR, KAIST, and DroneVehicle, notably $mAP=66.1$ on LLVIP with about 12.1M trainable parameters, while significantly reducing fine-tuning costs. The combination of explicit structural priors and semantic environmental reasoning yields robust performance across all-weather, cluttered, and aerial scenarios, with practical implications for efficient deployment on resource-constrained platforms.

Abstract

Multimodal object detection leveraging RGB and Infrared (IR) images is pivotal for robust perception in all-weather scenarios. While recent adapter-based approaches efficiently transfer RGB-pretrained foundation models to this task, they often prioritize model efficiency at the expense of cross-modal structural consistency. Consequently, critical structural cues are frequently lost when significant domain gaps arise, such as in high-contrast or nighttime environments. Moreover, conventional static multimodal fusion mechanisms typically lack environmental awareness, resulting in suboptimal adaptation and constrained detection performance under complex, dynamic scene variations. To address these limitations, we propose SLGNet, a parameter-efficient framework that synergizes hierarchical structural priors and language-guided modulation within a frozen Vision Transformer (ViT)-based foundation model. Specifically, we design a Structure-Aware Adapter to extract hierarchical structural representations from both modalities and dynamically inject them into the ViT to compensate for structural degradation inherent in ViT-based backbones. Furthermore, we propose a Language-Guided Modulation module that exploits VLM-driven structured captions to dynamically recalibrate visual features, thereby endowing the model with robust environmental awareness. Extensive experiments on the LLVIP, FLIR, KAIST, and DroneVehicle datasets demonstrate that SLGNet establishes new state-of-the-art performance. Notably, on the LLVIP benchmark, our method achieves an mAP of 66.1, while reducing trainable parameters by approximately 87% compared to traditional full fine-tuning. This confirms SLGNet as a robust and efficient solution for multimodal perception.

Figures (6)

• Figure 1: Comparison of multimodal adaptation paradigms: Existing strategies vs. our SLGNet. (a) Full Fine-tuning: Updates all parameters of the foundation model, leading to high computational costs and potential catastrophic forgetting. (b) Standard Adapter Tuning: Freezes the backbone and trains lightweight adapters. However, these methods often lack explicit structural constraints, leading to spatial detail loss. (c) SLGNet (Ours): We propose a synergistic framework that incorporates a Structure-Aware Adapter to preserve geometric details (bottom) and Language-Guided Modulation (top) to enhance semantic adaptability. The and icons indicate frozen and trainable parameters, respectively.
• Figure 2: Overview of the proposed SLGNet framework. The architecture synergizes a frozen Vision Transformer (ViT) backbone with two lightweight trainable modules: (1) the Structure-Aware Adapter (bottom), which extracts hierarchical structural priors from paired images via a Structure Encoder and injects them into ViT blocks using Feature Fusion Adapter (FF-Adapter); and (2) the Language-Guided Modulation (LGM) (right), which utilizes VLM-generated structured captions (Environment, Scene, Objects, Thermal) to recalibrate the final feature map via affine transformations ($\gamma, \beta$). The and icons indicate frozen and trainable parameters, respectively.
• Figure 3: Detailed architecture of the Structure Encoder. (a) The encoder employs progressive convolutional stages to extract hierarchical structural priors across multiple resolutions. (b) The Hierarchical Structural Alignment (HSA) module. It establishes a reference structural map $\nabla_{\text{ref}}$ and utilizes an SSIM-driven mechanism to dynamically weight multimodal features based on their hierarchical structural consistency.
• Figure 4: Validation mAP curves over training epochs on the FLIR dataset. The solid lines represent the mean mAP, while the shaded regions indicate the standard deviation range. The blue curve (Adapter-tuning) demonstrates faster convergence and higher stability (narrower error band) compared to the red curve (Full-tuning), confirming the robustness of our optimization strategy.
• Figure 5: Visualization of the structural feature learning process. (a)-(b) Input RGB and IR images. (c) The fused reference structure map ($\nabla_{\text{ref}}$). (d)-(f) Cosine similarity maps computed between the feature of the query patch (marked as $\bullet$) and all other patches in the adapted ViT output. The high similarity spreading coherently along structural boundaries (e.g., the pedestrian in (d) and the unannotated street light in (f)) demonstrates that the SA-Adapter effectively injects structural priors into the semantic feature space.