Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bringing RGB and IR Together: Hierarchical Multi-Modal Enhancement for Robust Transmission Line Detection

Shengdong Zhang, Xiaoqin Zhang, Wenqi Ren, Linlin Shen, Shaohua Wan, Jun Zhang, Yujing M Jiang

TL;DR

This work tackles robust transmission line detection from multimodal RGB and IR imagery by introducing HMMEN, a hierarchical fusion framework that uses Mutual Multi-Modal Enhanced Blocks (MMEB) to mutually strengthen RGB and IR features and Feature Alignment Blocks (FAB) to align multi-scale feature maps via deformable convolutions. The approach addresses misalignment at both input and feature levels, leveraging lightweight MobileNet backbones for edge-friendly deployment. Through a designed loss combining Binary Cross-Entropy and Dice terms, and extensive ablations on a RGB–IR UAV dataset, HMMEN demonstrates superior boundary precision and reduced false positives under diverse weather and lighting. The results indicate strong potential for practical large-scale UAV inspections in rural power grids, delivering robust TL detection across challenging conditions.

Abstract

Ensuring a stable power supply in rural areas relies heavily on effective inspection of power equipment, particularly transmission lines (TLs). However, detecting TLs from aerial imagery can be challenging when dealing with misalignments between visible light (RGB) and infrared (IR) images, as well as mismatched high- and low-level features in convolutional networks. To address these limitations, we propose a novel Hierarchical Multi-Modal Enhancement Network (HMMEN) that integrates RGB and IR data for robust and accurate TL detection. Our method introduces two key components: (1) a Mutual Multi-Modal Enhanced Block (MMEB), which fuses and enhances hierarchical RGB and IR feature maps in a coarse-to-fine manner, and (2) a Feature Alignment Block (FAB) that corrects misalignments between decoder outputs and IR feature maps by leveraging deformable convolutions. We employ MobileNet-based encoders for both RGB and IR inputs to accommodate edge-computing constraints and reduce computational overhead. Experimental results on diverse weather and lighting conditionsfog, night, snow, and daytimedemonstrate the superiority and robustness of our approach compared to state-of-the-art methods, resulting in fewer false positives, enhanced boundary delineation, and better overall detection performance. This framework thus shows promise for practical large-scale power line inspections with unmanned aerial vehicles.

Bringing RGB and IR Together: Hierarchical Multi-Modal Enhancement for Robust Transmission Line Detection

TL;DR

This work tackles robust transmission line detection from multimodal RGB and IR imagery by introducing HMMEN, a hierarchical fusion framework that uses Mutual Multi-Modal Enhanced Blocks (MMEB) to mutually strengthen RGB and IR features and Feature Alignment Blocks (FAB) to align multi-scale feature maps via deformable convolutions. The approach addresses misalignment at both input and feature levels, leveraging lightweight MobileNet backbones for edge-friendly deployment. Through a designed loss combining Binary Cross-Entropy and Dice terms, and extensive ablations on a RGB–IR UAV dataset, HMMEN demonstrates superior boundary precision and reduced false positives under diverse weather and lighting. The results indicate strong potential for practical large-scale UAV inspections in rural power grids, delivering robust TL detection across challenging conditions.

Abstract

Ensuring a stable power supply in rural areas relies heavily on effective inspection of power equipment, particularly transmission lines (TLs). However, detecting TLs from aerial imagery can be challenging when dealing with misalignments between visible light (RGB) and infrared (IR) images, as well as mismatched high- and low-level features in convolutional networks. To address these limitations, we propose a novel Hierarchical Multi-Modal Enhancement Network (HMMEN) that integrates RGB and IR data for robust and accurate TL detection. Our method introduces two key components: (1) a Mutual Multi-Modal Enhanced Block (MMEB), which fuses and enhances hierarchical RGB and IR feature maps in a coarse-to-fine manner, and (2) a Feature Alignment Block (FAB) that corrects misalignments between decoder outputs and IR feature maps by leveraging deformable convolutions. We employ MobileNet-based encoders for both RGB and IR inputs to accommodate edge-computing constraints and reduce computational overhead. Experimental results on diverse weather and lighting conditionsfog, night, snow, and daytimedemonstrate the superiority and robustness of our approach compared to state-of-the-art methods, resulting in fewer false positives, enhanced boundary delineation, and better overall detection performance. This framework thus shows promise for practical large-scale power line inspections with unmanned aerial vehicles.
Paper Structure (22 sections, 3 equations, 7 figures, 5 tables)

This paper contains 22 sections, 3 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transmission Line Detection
  4. Deep Learning Attention
  5. Feature Fusion
  6. Hierarchical Multi-Modal TLD
  7. Motivations
  8. Mutual Multi-modal Enhanced Block (MMEB)
  9. Feature Alignment Block (FAB)
  10. Training Losses
  11. Binary Cross-Entropy Loss
  12. Addressing Class Imbalance
  13. Dice Loss
  14. Final Loss Function
  15. Experimental Results
  16. ...and 7 more sections

Figures (7)

  • Figure 1: Architecture of the proposed Hierarchical Mutual Multi-modal Enhanced Network (HMMEN). The proposed model consists of DB, MMEB, FAB, and DB. All outputted features from the MMEB are passed to the FAB at the same scale. The FAB at the lowest resolution only aligns the features from MMEB.
  • Figure 2: A visual example of matched RGB and IR images alongside their corresponding ground truths (GTs). The first row displays the RGB images, the second row shows the IR images, and the third row presents the GTs.
  • Figure 3: Architecture of the proposed Mutual Multi-modal Enhanced Block (MMEB), illustrating the process of feature extraction, weight map calculation, and mutual enhancement for RGB and IR modalities.
  • Figure 4: Architecture of the proposed Feature Alignment Block (FAB), detailing the alignment process for upsampled high-level features and high-resolution low-level features to ensure coherent feature integration.
  • Figure 5: Visual examples of transmission line (TL) detection results using various methods. Areas circled in red highlight key differences and details for comparison.
  • ...and 2 more figures