Heuristical Comparison of Vision Transformers Against Convolutional Neural Networks for Semantic Segmentation on Remote Sensing Imagery

TL;DR

This work compares Vision Transformers against CNNs for semantic segmentation in remote sensing on the iSAID dataset. It introduces a novel combined weighted loss for CNNs and evaluates transfer learning with MaskFormer (ViT) versus training from scratch with UNet (CNN), focusing on accuracy (mIoU/Dice) and efficiency (inference time, FLOPs). Findings indicate the CNN with the proposed loss can achieve competitive performance with substantially fewer parameters, while ViT-based MaskFormer can yield higher end metrics at greater computational cost. The study provides practical guidance on architecture selection under resource constraints and highlights the trade-offs between throughput and segmentation quality in remote sensing applications.

Abstract

Vision Transformers (ViT) have recently brought a new wave of research in the field of computer vision. These models have performed particularly well in image classification and segmentation. Research on semantic and instance segmentation has accelerated with the introduction of the new architecture, with over 80% of the top 20 benchmarks for the iSAID dataset based on either the ViT architecture or the attention mechanism behind its success. This paper focuses on the heuristic comparison of three key factors of using (or not using) ViT for semantic segmentation of remote sensing aerial images on the iSAID dataset. The experimental results observed during this research were analyzed based on three objectives. First, we studied the use of a weighted fused loss function to maximize the mean Intersection over Union (mIoU) score and Dice score while minimizing entropy or class representation loss. Second, we compared transfer learning on Meta's MaskFormer, a ViT-based semantic segmentation model, against a generic UNet Convolutional Neural Network (CNN) based on mIoU, Dice scores, training efficiency, and inference time. Third, we examined the trade-offs between the two models in comparison to current state-of-the-art segmentation models. We show that the novel combined weighted loss function significantly boosts the CNN model's performance compared to transfer learning with ViT. The code for this implementation can be found at: https://github.com/ashimdahal/ViT-vs-CNN-Image-Segmentation.

Figures (7)

• Figure 1: Brief Overview of the Training and Validation Lifecycle
• Figure 2: Custom UNet CNN architecture
• Figure 3: MaskFormer ViT architecture from maskformer
• Figure 4: Comparison of Metrics over Epochs
• Figure 5: Sample Visualization of model's output