Semantic-Guided Natural Language and Visual Fusion for Cross-Modal Interaction Based on Tiny Object Detection

TL;DR

This work tackles tiny object detection under open-world conditions by integrating semantic information from natural language with visual features. It employs a BERT-based text encoder and a CNN-based PRB-FPN-Net to perform semantic-guided cross-modal fusion, with lemmatization and keyword mapping to align language and vision. The approach introduces a parallel Lead/Aux fusion head design and backbone variants (ELAN, MSP, CSP) to improve multi-scale detection efficiency. Empirical results on COCO and Objects365 demonstrate strong AP (e.g., $AP_{COCO}=52.6%$) while using substantially fewer parameters than Transformer-heavy baselines, indicating practical viability for resource-constrained deployments. This work highlights language-guided perception as a path to robust cross-modal tiny-object recognition and adaptability to real-world challenges.

Abstract

This paper introduces a cutting-edge approach to cross-modal interaction for tiny object detection by combining semantic-guided natural language processing with advanced visual recognition backbones. The proposed method integrates the BERT language model with the CNN-based Parallel Residual Bi-Fusion Feature Pyramid Network (PRB-FPN-Net), incorporating innovative backbone architectures such as ELAN, MSP, and CSP to optimize feature extraction and fusion. By employing lemmatization and fine-tuning techniques, the system aligns semantic cues from textual inputs with visual features, enhancing detection precision for small and complex objects. Experimental validation using the COCO and Objects365 datasets demonstrates that the model achieves superior performance. On the COCO2017 validation set, it attains a 52.6% average precision (AP), outperforming YOLO-World significantly while maintaining half the parameter consumption of Transformer-based models like GLIP. Several test on different of backbones such ELAN, MSP, and CSP further enable efficient handling of multi-scale objects, ensuring scalability and robustness in resource-constrained environments. This study underscores the potential of integrating natural language understanding with advanced backbone architectures, setting new benchmarks in object detection accuracy, efficiency, and adaptability to real-world challenges.

Figures (5)

• Figure 1: The proposed methodology framework outlined in this research
• Figure 2: Overall Framework and Workflow
• Figure 3: Experimental Results Comparison 1
• Figure 4: Experimental Results Comparison 2
• Figure 5: Experimental Results Comparison 3