MSFNet-CPD: Multi-Scale Cross-Modal Fusion Network for Crop Pest Detection

TL;DR

The paper tackles crop pest detection under challenging real-world conditions by introducing MSFNet-CPD, a multi-scale cross-modal fusion network that combines image features with textual descriptions. It incorporates a super-resolution module (LSRGAN), a text-image fusion pipeline (ITF) with an image-text converter (ITC), and a pest-target identification head, all trained on IP102-derived multi-modal datasets CTIP102, STIP102, HIP102, and MTIP102 created via ACIE augmentation. Key contributions include the first integration of high-frequency image information with text for pest detection, the construction of multi-modal benchmarks, and comprehensive ablations showing the value of text and SR components. The results demonstrate significant gains over unimodal baselines and state-of-the-art detectors, with practical implications for robust pest monitoring in diverse agricultural environments.

Abstract

Accurate identification of agricultural pests is essential for crop protection but remains challenging due to the large intra-class variance and fine-grained differences among pest species. While deep learning has advanced pest detection, most existing approaches rely solely on low-level visual features and lack effective multi-modal integration, leading to limited accuracy and poor interpretability. Moreover, the scarcity of high-quality multi-modal agricultural datasets further restricts progress in this field. To address these issues, we construct two novel multi-modal benchmarks-CTIP102 and STIP102-based on the widely-used IP102 dataset, and introduce a Multi-scale Cross-Modal Fusion Network (MSFNet-CPD) for robust pest detection. Our approach enhances visual quality via a super-resolution reconstruction module, and feeds both the original and reconstructed images into the network to improve clarity and detection performance. To better exploit semantic cues, we propose an Image-Text Fusion (ITF) module for joint modeling of visual and textual features, and an Image-Text Converter (ITC) that reconstructs fine-grained details across multiple scales to handle challenging backgrounds. Furthermore, we introduce an Arbitrary Combination Image Enhancement (ACIE) strategy to generate a more complex and diverse pest detection dataset, MTIP102, improving the model's generalization to real-world scenarios. Extensive experiments demonstrate that MSFNet-CPD consistently outperforms state-of-the-art methods on multiple pest detection benchmarks. All code and datasets will be made publicly available at: https://github.com/Healer-ML/MSFNet-CPD.

Figures (6)

• Figure 1: MSFNet-CPD Model Architecture (A. LSRGAN for super-resolution of low-quality images, B. Picture to ITF Transformer (TIC), C. Multi-scale Cross-modal Fusion ITF Module, D. ITF to Neck Network Converter (ITC), and E. Pest Target Identification (PTI). Numbers represent categories in Original Image and Pre Image, numbers represent class confidence and category).
• Figure 2: ITF Specific Process. ($B_i$, $C_i$, $V_i$ stand for different scale image features and $T$ stands for text features).
• Figure 3: Partial Presentation of Multi-modal Dataset.
• Figure 4: Visualization of the semantic correlation analysis of simple and complex text descriptions. (Horizontal and vertical coordinates in the figure indicate the coordinates after mapping the feature vectors into a low-dimensional space).
• Figure 5: Comparison of Indicators for Different Data Sets.