Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FTPFusion: Frequency-Aware Infrared and Visible Video Fusion with Temporal Perturbation

Xilai Li, Chusheng Fang, Xiaosong Li

Abstract

Infrared and visible video fusion plays a critical role in intelligent surveillance and low-light monitoring. However, maintaining temporal stability while preserving spatial detail remains a fundamental challenge. Existing methods either focus on frame-wise enhancement with limited temporal modeling or rely on heavy spatio-temporal aggregation that often sacrifices high-frequency details. In this paper, we propose FTPFusion, a frequency-aware infrared and visible video fusion method based on temporal perturbation and sparse cross-modal interaction. Specifically, FTPFusion decomposes the feature representations into high-frequency and low-frequency components for collaborative modeling. The high-frequency branch performs sparse cross-modal spatio-temporal interaction to capture motion-related context and complementary details. The low-frequency branch introduces a temporal perturbation strategy to enhance robustness against complex video variations, such as flickering, jitter, and local misalignment. Furthermore, we design an offset-aware temporal consistency constraint to explicitly stabilize cross-frame representations under temporal disturbances. Extensive experiments on multiple public benchmarks demonstrate that FTPFusion consistently outperforms state-of-the-art methods across multiple metrics in both spatial fidelity and temporal consistency. The source code will be available at https://github.com/ixilai/FTPFusion.

FTPFusion: Frequency-Aware Infrared and Visible Video Fusion with Temporal Perturbation

Abstract

Infrared and visible video fusion plays a critical role in intelligent surveillance and low-light monitoring. However, maintaining temporal stability while preserving spatial detail remains a fundamental challenge. Existing methods either focus on frame-wise enhancement with limited temporal modeling or rely on heavy spatio-temporal aggregation that often sacrifices high-frequency details. In this paper, we propose FTPFusion, a frequency-aware infrared and visible video fusion method based on temporal perturbation and sparse cross-modal interaction. Specifically, FTPFusion decomposes the feature representations into high-frequency and low-frequency components for collaborative modeling. The high-frequency branch performs sparse cross-modal spatio-temporal interaction to capture motion-related context and complementary details. The low-frequency branch introduces a temporal perturbation strategy to enhance robustness against complex video variations, such as flickering, jitter, and local misalignment. Furthermore, we design an offset-aware temporal consistency constraint to explicitly stabilize cross-frame representations under temporal disturbances. Extensive experiments on multiple public benchmarks demonstrate that FTPFusion consistently outperforms state-of-the-art methods across multiple metrics in both spatial fidelity and temporal consistency. The source code will be available at https://github.com/ixilai/FTPFusion.

Paper Structure

This paper contains 34 sections, 61 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-modality Image Fusion
  4. Multi-modality Video Fusion
  5. Methodology
  6. Overview
  7. Dual-Branch Frequency-Aware Fusion Module
  8. Low-Frequency Temporal Perturbation Enhancement Module
  9. Low-Frequency Shared Temporal Context Module
  10. SCAM: Sparse Cross-Modal Attention Module
  11. Loss Functions
  12. Offset-aware Temporal Consistency Loss
  13. Experiments
  14. Training Sets
  15. Comparison Methods and Datasets
  16. ...and 19 more sections

Figures (7)

  • Figure 1: Comparison of the Limitations of Two Mainstream Video Fusion Methods.
  • Figure 2: The overall framework of the proposed algorithm.
  • Figure 3: Qualitative comparison of all methods on the M3SVD, HDO and VTMOT datasets.
  • Figure 4: Visual comparison of the temporal stability of different methods.
  • Figure 5: Comparison of segmentation results of different fusion methods in the video segmentation task.
  • ...and 2 more figures