Cross-Modal Alignment and Fusion for RGB-D Transmission-Line Defect Detection

TL;DR

CMAFNet tackles the difficulty of identifying small, low-contrast transmission-line defects in UAV RGB-D imagery by introducing a purify-then-fuse paradigm. It combines a dual-branch RGB-D backbone with Semantic Recomposition Modules for intra-modal purification and a Contextual Semantic Integration Framework for cross-modal global context, enabling explicit distribution alignment and robust small-object detection. The approach yields state-of-the-art performance on the TL-RGBD benchmark (e.g., mAP@50 up to 0.322) while offering a lightweight variant that runs at real-time Framerates, demonstrating strong accuracy-efficiency trade-offs for field deployment. These contributions advance practical RGB-D fusion in aerial infrastructure inspection, providing improved reliability for maintenance planning and asset safety.

Abstract

Transmission line defect detection remains challenging for automated UAV inspection due to the dominance of small-scale defects, complex backgrounds, and illumination variations. Existing RGB-based detectors, despite recent progress, struggle to distinguish geometrically subtle defects from visually similar background structures under limited chromatic contrast. This paper proposes CMAFNet, a Cross-Modal Alignment and Fusion Network that integrates RGB appearance and depth geometry through a principled purify-then-fuse paradigm. CMAFNet consists of a Semantic Recomposition Module that performs dictionary-based feature purification via a learned codebook to suppress modality-specific noise while preserving defect-discriminative information, and a Contextual Semantic Integration Framework that captures global spatial dependencies using partial-channel attention to enhance structural semantic reasoning. Position-wise normalization within the purification stage enforces explicit reconstruction-driven cross-modal alignment, ensuring statistical compatibility between heterogeneous features prior to fusion. Extensive experiments on the TLRGBD benchmark, where 94.5% of instances are small objects, demonstrate that CMAFNet achieves 32.2% mAP@50 and 12.5% APs, outperforming the strongest baseline by 9.8 and 4.0 percentage points, respectively. A lightweight variant reaches 24.8% mAP50 at 228 FPS with only 4.9M parameters, surpassing all YOLO-based detectors while matching transformer-based methods at substantially lower computational cost.

Figures (10)

• Figure 1: Overall architecture of the proposed Cross-Modal Alignment and Fusion Network. The dual-branch encoder independently processes RGB and depth modalities through symmetric pathways, with Semantic Recomposition Modules deployed at $P_3$ and $P_4$ scales for intra-modal semantic purification and cross-modal distribution alignment. Cross-modal fusion is performed at $P_4$ and $P_5$ levels, where the Contextual Semantic Integration Framework provides global attention-based context modeling at the deepest semantic scale. The detection head implements bidirectional feature propagation through FPN and PAN pathways for multi-scale defect detection.
• Figure 2: Architecture of the Semantic Recomposition Module. The input features are projected into a dictionary space via pointwise convolution, refined through depthwise spatial convolution, normalized at each spatial position (PONO denotes position-wise normalization), and decoded back to the original channel dimension. The output combines the refined features with the input through learnable residual mixing.
• Figure 3: Architecture of the Contextual Semantic Integration Framework. The input features pass through a convolutional stem that harmonizes channel dimensions, followed by $n$ stacked Contextual Semantic Interaction Blocks that perform attention-based global modeling with interleaved normalization. An output projection aligns the refined features to the target dimension.
• Figure 4: Structure of the Contextual Semantic Interaction Block. Each block comprises a multi-head self-attention layer followed by ASRM normalization, then a feed-forward network followed by another ASRM normalization. Residual connections facilitate gradient flow at both stages.
• Figure 5: Sample image pairs from TL-RGBD dataset. (a) and (c) are RGB images captured by UAV-mounted cameras showing transmission line components under real inspection conditions. (b) and (d) are the corresponding depth maps acquired synchronously, providing geometric information for defect localization.