Strip-Fusion: Spatiotemporal Fusion for Multispectral Pedestrian Detection

TL;DR

Strip-Fusion tackles multispectral pedestrian detection by introducing a spatiotemporal fusion network that remains robust to misalignment and lighting changes. It combines a Strip Fusion Module with temporally adaptive convolutions (TAdaConv) and a KL-divergence loss to balance RGB and thermal information, along with a dedicated post-processing step to reduce false positives. The method yields competitive results on KAIST and CVC-14, with notable gains under heavy occlusion and misalignment, and demonstrates practicality with reasonable inference speeds. Overall, Strip-Fusion advances robust, temporally informed multispectral fusion for pedestrian detection in dynamic, real-world scenarios.

Abstract

Pedestrian detection is a critical task in robot perception. Multispectral modalities (visible light and thermal) can boost pedestrian detection performance by providing complementary visual information. Several gaps remain with multispectral pedestrian detection methods. First, existing approaches primarily focus on spatial fusion and often neglect temporal information. Second, RGB and thermal image pairs in multispectral benchmarks may not always be perfectly aligned. Pedestrians are also challenging to detect due to varying lighting conditions, occlusion, etc. This work proposes Strip-Fusion, a spatial-temporal fusion network that is robust to misalignment in input images, as well as varying lighting conditions and heavy occlusions. The Strip-Fusion pipeline integrates temporally adaptive convolutions to dynamically weigh spatial-temporal features, enabling our model to better capture pedestrian motion and context over time. A novel Kullback-Leibler divergence loss was designed to mitigate modality imbalance between visible and thermal inputs, guiding feature alignment toward the more informative modality during training. Furthermore, a novel post-processing algorithm was developed to reduce false positives. Extensive experimental results show that our method performs competitively for both the KAIST and the CVC-14 benchmarks. We also observed significant improvements compared to previous state-of-the-art on challenging conditions such as heavy occlusion and misalignment.

Figures (5)

• Figure 1: High-level overview of Strip-Fusion. Input: Sequence of visible–thermal image pairs, each image treated as a 3-channel input. Output: Detected pedestrian bounding boxes at the final frame. Strip-Fusion combines visible and thermal features at multiple scales: 80$\times$80, 40$\times$40, and 20$\times$20.
• Figure 2: Our proposed strip fusion module for spatial and temporal fusion. Best viewed in color.
• Figure 3: (a) Given input feature $X_{m_1}$$\in$$\mathbb{R}^{ 2H \times W \times C/2}$, the CGSFMM splits the feature into $N$ patches along the channel direction, and applies Strip-MLP along the row and column direction in series. (b) Given input feature $X_{m_2}$$\in$$\mathbb{R}^{ 2H \times W \times C/2}$, the LSFMM applies a row and column Strip-MLP, combines their outputs, processes the result through an MLP, performs global average pooling, and finally uses a softmax layer to weight the contributions of the row and column components along with the original input. (c) The channel mixing block integrates information across channels and patches. GRN refers to Global Response Normalization.
• Figure 4: Visual examples of KAIST detection results for heavily occluded pedestrians. Green and yellow bounding boxes correspond to ground truth and detection results, respectively. Ours show better detection on occluded/small pedestrians. Note that the KAIST dataset contains well aligned examples so we show only the visible images (See Multimedia Fig. R1).
• Figure 5: Visual results on the CVC-14 dataset for two misaligned image pairs. Numbers indicate detection confidence scores. The top row shows that our Algo.1 post-processing approach reduces false positives by removing low-confidence and non-overlapping detections, leading to more consistent performance. The bottom row illustrates a "failure" case where high-confidence false positives persist after post-processing in areas of misalignment. Detections below 0.2 confidence are not shown.