SFFR: Spatial-Frequency Feature Reconstruction for Multispectral Aerial Object Detection

TL;DR

This paper tackles multispectral aerial object detection by addressing the underexplored frequency-domain information and severe scale variation inherent in UAV imagery. It introduces Spatial-Frequency Feature Reconstruction (SFFR), which leverages Kolmogorov–Arnold Network (KAN) to reconstruct complementary spatial and frequency representations before fusion, via the Frequency Component Exchange KAN (FCEKAN) and Multi-Scale Gaussian KAN (MSGKAN) modules. The approach integrates intra-modal nonlinear spatial modeling with cross-modal frequency-aware exchange and achieves robust cross-modal fusion and scale-adaptive representation, validated on SeaDroneSee, DroneVehicle, and DVTOD with state-of-the-art results and thorough ablations. The work demonstrates that explicit spatial-frequency reconstruction enhances cross-modal alignment and detection robustness, offering a practical framework for UAV multispectral perception and potential real-time deployment with further efficiency optimizations.

Abstract

Recent multispectral object detection methods have primarily focused on spatial-domain feature fusion based on CNNs or Transformers, while the potential of frequency-domain feature remains underexplored. In this work, we propose a novel Spatial and Frequency Feature Reconstruction method (SFFR) method, which leverages the spatial-frequency feature representation mechanisms of the Kolmogorov-Arnold Network (KAN) to reconstruct complementary representations in both spatial and frequency domains prior to feature fusion. The core components of SFFR are the proposed Frequency Component Exchange KAN (FCEKAN) module and Multi-Scale Gaussian KAN (MSGKAN) module. The FCEKAN introduces an innovative selective frequency component exchange strategy that effectively enhances the complementarity and consistency of cross-modal features based on the frequency feature of RGB and IR images. The MSGKAN module demonstrates excellent nonlinear feature modeling capability in the spatial domain. By leveraging multi-scale Gaussian basis functions, it effectively captures the feature variations caused by scale changes at different UAV flight altitudes, significantly enhancing the model's adaptability and robustness to scale variations. It is experimentally validated that our proposed FCEKAN and MSGKAN modules are complementary and can effectively capture the frequency and spatial semantic features respectively for better feature fusion. Extensive experiments on the SeaDroneSee, DroneVehicle and DVTOD datasets demonstrate the superior performance and significant advantages of the proposed method in UAV multispectral object perception task. Code will be available at https://github.com/qchenyu1027/SFFR.

Figures (8)

• Figure 1: Examples of RGB–IR image pairs from the DVTOD dataset. The first and second rows show RGB and IR images respectively. RGB images excel at capturing color and texture information through visible light reflection, whereas IR images specialize in visualizing heat radiation. The two modalities complement each other, especially in challenging conditions such as low illumination, occlusion, or complex backgrounds, thereby enhancing object visibility and discriminability.
• Figure 2: Overview of the proposed multispectral aerial object detection framework. (The upper and lower branches extract RGB and IR features, respectively. Cross-modal and multi-scale fusion is performed via the proposed KANFUSION module. Subsequently, a multi-scale fusion network in the Neck aggregates and enhances the fused features, and the Detection Head produces the final detection results.)
• Figure 3: Illustration of our cross-modal feature fusion module, which consists of two parallel feature extraction networks. The overall fusion module KANFUSION (a) integrates RGB and IR features via the SFFR (b) module, whose detailed architecture is shown on the right.
• Figure 4: Overview of the proposed five structures: from left to right are the Intra-modality feature enhancement module, the cross-modal feature enhancement module, the FCEKAN module, and finally the SFFR module, which constitutes our final method.
• Figure 5: Visualization results with different gridsizes. The original image is presented in the leftmost panel, followed by the feature maps generated at gridsizes of 4, 8, 16, 32, and 64, arranged sequentially from left to right.