Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction

Xilai Li, Weijun Jiang, Xiaosong Li, Yang Liu, Hongbin Wang, Tao Ye, Huafeng Li, Haishu Tan

Abstract

Infrared and visible video fusion combines the object saliency from infrared images with the texture details from visible images to produce semantically rich fusion results. However, most existing methods are designed for static image fusion and cannot effectively handle frame-to-frame motion in videos. Current video fusion methods improve temporal consistency by introducing interactions across frames, but they often require high computational cost. To mitigate these challenges, we propose MAVFusion, an end-to-end video fusion framework featuring a motion-aware sparse interaction mechanism that enhances efficiency while maintaining superior fusion quality. Specifically, we leverage optical flow to identify dynamic regions in multi-modal sequences, adaptively allocating computationally intensive cross-modal attention to these sparse areas to capture salient transitions and facilitate inter-modal information exchange. For static background regions, a lightweight weak interaction module is employed to maintain structural and appearance integrity. By decoupling the processing of dynamic and static regions, MAVFusion simultaneously preserves temporal consistency and fine-grained details while significantly accelerating inference. Extensive experiments demonstrate that MAVFusion achieves state-of-the-art performance on multiple infrared and visible video benchmarks, achieving a speed of 14.16\,FPS at $640 \times 480$ resolution. The source code will be available at https://github.com/ixilai/MAVFusion.

MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction

Abstract

Infrared and visible video fusion combines the object saliency from infrared images with the texture details from visible images to produce semantically rich fusion results. However, most existing methods are designed for static image fusion and cannot effectively handle frame-to-frame motion in videos. Current video fusion methods improve temporal consistency by introducing interactions across frames, but they often require high computational cost. To mitigate these challenges, we propose MAVFusion, an end-to-end video fusion framework featuring a motion-aware sparse interaction mechanism that enhances efficiency while maintaining superior fusion quality. Specifically, we leverage optical flow to identify dynamic regions in multi-modal sequences, adaptively allocating computationally intensive cross-modal attention to these sparse areas to capture salient transitions and facilitate inter-modal information exchange. For static background regions, a lightweight weak interaction module is employed to maintain structural and appearance integrity. By decoupling the processing of dynamic and static regions, MAVFusion simultaneously preserves temporal consistency and fine-grained details while significantly accelerating inference. Extensive experiments demonstrate that MAVFusion achieves state-of-the-art performance on multiple infrared and visible video benchmarks, achieving a speed of 14.16\,FPS at resolution. The source code will be available at https://github.com/ixilai/MAVFusion.

Paper Structure

This paper contains 34 sections, 10 equations, 10 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Infrared and Visible Image Fusion
  4. Infrared and Visible Video Fusion
  5. Methodology
  6. Motion-Aware Feature Alignment Module
  7. Motion-Guided Dual-Interaction Module
  8. Dynamic Branch: Sparse Attention in Dynamic Regions
  9. Static Branch: Weak Interaction in Static Regions
  10. Adaptive Reconstruction and Fusion of Dynamic and Static Features
  11. Loss Function
  12. Spatial Fidelity Constraint
  13. Temporal Consistency Constraint
  14. Experiments
  15. Experiment Settings
  16. ...and 19 more sections

Figures (10)

  • Figure 1: Comparison of the proposed algorithm with existing video fusion method r122 in terms of multi-modal interaction strategies and computational efficiency.
  • Figure 2: The overall framework of the proposed algorithm.
  • Figure 3: Effect of Global Strong Interaction on Motion Clues and Structural Integrity.
  • Figure 4: Qualitative comparison of all methods on the M3SVD dataset.
  • Figure 5: Qualitative comparison of all methods on the HDO dataset.
  • ...and 5 more figures