Multimodal Spatio-temporal Graph Learning for Alignment-free RGBT Video Object Detection

TL;DR

This work tackles alignment-free RGB-Thermal Video Object Detection by introducing MSGNet, a graph-based fusion framework that operates without precise pixel-level cross-modal alignment. It combines an Adaptive Partitioning Layer (APL) for coarse spatial alignment, a Spatial Sparse Graph Learning Module (S-SGLM) for robust cross-modal fusion, and Hybrid Structured Temporal Modeling (HSTM) to capture meaningful temporal dynamics while suppressing redundancy. The approach demonstrates strong performance on both aligned VT-VOD50 and unaligned UVT-VOD2024 benchmarks, outperforming existing methods with favorable speed and parameter efficiency. By enabling robust multimodal fusion across varying spatial distributions and temporal contexts, MSGNet offers a practical, scalable solution for real-world RGBT VOD tasks with alignment challenges.

Abstract

RGB-Thermal Video Object Detection (RGBT VOD) can address the limitation of traditional RGB-based VOD in challenging lighting conditions, making it more practical and effective in many applications. However, similar to most RGBT fusion tasks, it still mainly relies on manually aligned multimodal image pairs. In this paper, we propose a novel Multimodal Spatio-temporal Graph learning Network (MSGNet) for alignment-free RGBT VOD problem by leveraging the robust graph representation learning model. Specifically, we first design an Adaptive Partitioning Layer (APL) to estimate the corresponding regions of the Thermal image within the RGB image (high-resolution), achieving a preliminary inexact alignment. Then, we introduce the Spatial Sparse Graph Learning Module (S-SGLM) which employs a sparse information passing mechanism on the estimated inexact alignment to achieve reliable information interaction between different modalities. Moreover, to fully exploit the temporal cues for RGBT VOD problem, we introduce Hybrid Structured Temporal Modeling (HSTM), which involves a Temporal Sparse Graph Learning Module (T-SGLM) and Temporal Star Block (TSB). T-SGLM aims to filter out some redundant information between adjacent frames by employing the sparse aggregation mechanism on the temporal graph. Meanwhile, TSB is dedicated to achieving the complementary learning of local spatial relationships. Extensive comparative experiments conducted on both the aligned dataset VT-VOD50 and the unaligned dataset UVT-VOD2024 demonstrate the effectiveness and superiority of our proposed method. Our project will be made available on our website for free public access.

Figures (8)

• Figure 1: Owing to disparities in sensor field of view and resolution, RGBT image pairs directly captured by sensors in real-world scenarios typically resemble the situation depicted in (c), where the two modalities are not strictly spatially aligned. Previous studies often necessitate spatially aligned or near-aligned image pairs, as shown in (a) and (b), as input. However, achieving such alignment typically requires manual intervention, involving the use of calibration data or specialized algorithms for preprocessing, which can be cumbersome and impractical for specific applications.
• Figure 2: The schematic diagram of MSGNet. We utilize a total of four input frames. MSGNet initially employs APL and S-SGLM to address the challenge of alignment-free fusion across disparate modalities. Subsequently, HSTM is leveraged to encapsulate the temporal features at two distinct time points. Ultimately, the detection head processes these enriched features to yield the final prediction.
• Figure 3: The structure of APL and S-SGLM. The APL module first jointly predicts the feature maps of RGB and Thermal images. To achieve this, we employ a progressive approach to derive a value $\lambda$, which serves as an intermediate representation. Subsequently, we apply our predefined rules to compute a value $\gamma$ based on $\lambda$. The value $\gamma$ is utilized to identify and select a local region within the RGB image that corresponds to a larger field of view. This region is hypothesized to spatially align with the area of interest in the thermal image. The subsequent interaction between the RGB and thermal image features is then facilitated through the composition and aggregation of information from these two corresponding regions. This process enables the model to effectively integrate multimodal data while preserving spatial coherence between the RGB and thermal domains. Among them, "Proj." denotes the projection layer implemented with a convolutional kernel of size $1 \times 1$.
• Figure 4: Schematic diagram of the image encoder used by our method. The encoder accepts the input image and produces three layers of feature maps with distinct scales. They have both shallow detail information and high-level semantic information. These feature maps are subsequently transmitted to the intermediate stage of the network for feature fusion.
• Figure 5: The structure of TSB. The fundamental components of the TSB are convolutional layers and linear layers. To enhance computational efficiency, we strategically intersperse element-wise multiplication operations, which rapidly project features into new high-dimensional subspaces. This design choice allows for swift and effective feature transformation while maintaining minimal computational overhead.