Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiRecNetV2: A Transformer-Enhanced Network for Aerial Disaster Recognition

Demetris Shianios, Panayiotis Kolios, Christos Kyrkou

TL;DR

DiRecNetV2 is introduced, an improved hybrid model that utilizes convolutional and transformer layers that merges the inductive biases of CNNs for robust feature extraction with the global context understanding of Transformers, maintaining a low computational load ideal for UAV applications.

Abstract

The integration of Unmanned Aerial Vehicles (UAVs) with artificial intelligence (AI) models for aerial imagery processing in disaster assessment, necessitates models that demonstrate exceptional accuracy, computational efficiency, and real-time processing capabilities. Traditionally Convolutional Neural Networks (CNNs), demonstrate efficiency in local feature extraction but are limited by their potential for global context interpretation. On the other hand, Vision Transformers (ViTs) show promise for improved global context interpretation through the use of attention mechanisms, although they still remain underinvestigated in UAV-based disaster response applications. Bridging this research gap, we introduce DiRecNetV2, an improved hybrid model that utilizes convolutional and transformer layers. It merges the inductive biases of CNNs for robust feature extraction with the global context understanding of Transformers, maintaining a low computational load ideal for UAV applications. Additionally, we introduce a new, compact multi-label dataset of disasters, to set an initial benchmark for future research, exploring how models trained on single-label data perform in a multi-label test set. The study assesses lightweight CNNs and ViTs on the AIDERSv2 dataset, based on the frames per second (FPS) for efficiency and the weighted F1 scores for classification performance. DiRecNetV2 not only achieves a weighted F1 score of 0.964 on a single-label test set but also demonstrates adaptability, with a score of 0.614 on a complex multi-label test set, while functioning at 176.13 FPS on the Nvidia Orin Jetson device.

DiRecNetV2: A Transformer-Enhanced Network for Aerial Disaster Recognition

TL;DR

DiRecNetV2 is introduced, an improved hybrid model that utilizes convolutional and transformer layers that merges the inductive biases of CNNs for robust feature extraction with the global context understanding of Transformers, maintaining a low computational load ideal for UAV applications.

Abstract

The integration of Unmanned Aerial Vehicles (UAVs) with artificial intelligence (AI) models for aerial imagery processing in disaster assessment, necessitates models that demonstrate exceptional accuracy, computational efficiency, and real-time processing capabilities. Traditionally Convolutional Neural Networks (CNNs), demonstrate efficiency in local feature extraction but are limited by their potential for global context interpretation. On the other hand, Vision Transformers (ViTs) show promise for improved global context interpretation through the use of attention mechanisms, although they still remain underinvestigated in UAV-based disaster response applications. Bridging this research gap, we introduce DiRecNetV2, an improved hybrid model that utilizes convolutional and transformer layers. It merges the inductive biases of CNNs for robust feature extraction with the global context understanding of Transformers, maintaining a low computational load ideal for UAV applications. Additionally, we introduce a new, compact multi-label dataset of disasters, to set an initial benchmark for future research, exploring how models trained on single-label data perform in a multi-label test set. The study assesses lightweight CNNs and ViTs on the AIDERSv2 dataset, based on the frames per second (FPS) for efficiency and the weighted F1 scores for classification performance. DiRecNetV2 not only achieves a weighted F1 score of 0.964 on a single-label test set but also demonstrates adaptability, with a score of 0.614 on a complex multi-label test set, while functioning at 176.13 FPS on the Nvidia Orin Jetson device.

Paper Structure

This paper contains 25 sections, 8 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Label Dataset and Hybrid CNN-ViT Model
  4. Multi-Disaster Dataset
  5. Single and Multi-Label Classification
  6. Lightweight Deep Learning Models
  7. Convolutional Neural Networks:
  8. Vision Transformers:
  9. Transformer Enhanced Convolutional Network Architecture
  10. DiRecNet Feature Extractor:
  11. Transformer Encoder Block:
  12. Training Process
  13. Datasets:
  14. Data Pre-Processing:
  15. Transfer Learning:
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Sample of images in the database depicting various multi-label disaster instances.
  • Figure 2: The DiRecNetV2 model architecture, showcasing how features extracted by CNN blocks are fed into the encoder blocks of ViTs of the Vision Transformer, significantly enhancing the model’s capabilities.
  • Figure 3: The examples demonstrate DiRecNetV2's proficiency in identifying diverse disaster situations. Using a subset of four test set images, these images show the model's robust classification accuracy for earthquakes, fires, floods, and normal cases.
  • Figure 4: Examples of images from the multi-label dataset showcase the predictions of DiRecNetV2, trained for multi-label scenarios. The predictions illustrate the model's accurate identification of dual instances, with probabilities exceeding 50% for two classes within the same image, underscoring its proficiency in handling complex multi-label classifications.