Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Near-Infrared and Low-Rank Adaptation of Vision Transformers in Remote Sensing

Irem Ulku, O. Ozgur Tanriover, Erdem Akagündüz

TL;DR

This work tackles semantic segmentation in Near-Infrared (NIR) remote sensing under limited labeled data and cross-domain transfer from RGB. It applies Low-Rank Adaptation (LoRA) to RGB-pretrained Vision Transformers (ViT) to adapt them to the NIR domain with minimal trainable parameters. Evaluations on DSTL and RIT-18 show that LoRA-enabled ViT-L/16 achieves the strongest NIR performance, while reducing trainable parameters by approximately 97.5% and outperforming RGB baselines in the NIR domain. The findings suggest that LoRA is a practical path for domain adaptation in multispectral remote sensing, with potential for extending to other segmentation heads and related tasks in the NIR domain.

Abstract

Plant health can be monitored dynamically using multispectral sensors that measure Near-Infrared reflectance (NIR). Despite this potential, obtaining and annotating high-resolution NIR images poses a significant challenge for training deep neural networks. Typically, large networks pre-trained on the RGB domain are utilized to fine-tune infrared images. This practice introduces a domain shift issue because of the differing visual traits between RGB and NIR images.As an alternative to fine-tuning, a method called low-rank adaptation (LoRA) enables more efficient training by optimizing rank-decomposition matrices while keeping the original network weights frozen. However, existing parameter-efficient adaptation strategies for remote sensing images focus on RGB images and overlook domain shift issues in the NIR domain. Therefore, this study investigates the potential benefits of using vision transformer (ViT) backbones pre-trained in the RGB domain, with low-rank adaptation for downstream tasks in the NIR domain. Extensive experiments demonstrate that employing LoRA with pre-trained ViT backbones yields the best performance for downstream tasks applied to NIR images.

Near-Infrared and Low-Rank Adaptation of Vision Transformers in Remote Sensing

TL;DR

This work tackles semantic segmentation in Near-Infrared (NIR) remote sensing under limited labeled data and cross-domain transfer from RGB. It applies Low-Rank Adaptation (LoRA) to RGB-pretrained Vision Transformers (ViT) to adapt them to the NIR domain with minimal trainable parameters. Evaluations on DSTL and RIT-18 show that LoRA-enabled ViT-L/16 achieves the strongest NIR performance, while reducing trainable parameters by approximately 97.5% and outperforming RGB baselines in the NIR domain. The findings suggest that LoRA is a practical path for domain adaptation in multispectral remote sensing, with potential for extending to other segmentation heads and related tasks in the NIR domain.

Abstract

Plant health can be monitored dynamically using multispectral sensors that measure Near-Infrared reflectance (NIR). Despite this potential, obtaining and annotating high-resolution NIR images poses a significant challenge for training deep neural networks. Typically, large networks pre-trained on the RGB domain are utilized to fine-tune infrared images. This practice introduces a domain shift issue because of the differing visual traits between RGB and NIR images.As an alternative to fine-tuning, a method called low-rank adaptation (LoRA) enables more efficient training by optimizing rank-decomposition matrices while keeping the original network weights frozen. However, existing parameter-efficient adaptation strategies for remote sensing images focus on RGB images and overlook domain shift issues in the NIR domain. Therefore, this study investigates the potential benefits of using vision transformer (ViT) backbones pre-trained in the RGB domain, with low-rank adaptation for downstream tasks in the NIR domain. Extensive experiments demonstrate that employing LoRA with pre-trained ViT backbones yields the best performance for downstream tasks applied to NIR images.
Paper Structure (10 sections, 10 equations, 3 figures, 3 tables)

This paper contains 10 sections, 10 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Model Architecture
  3. Encoder
  4. LoRA Adaptation
  5. Segmentation Head
  6. Experimental Results
  7. Image sets
  8. Experimental Setup
  9. Results and Discussions
  10. Conclusion

Figures (3)

  • Figure 1: The proposed LoRA-based ViT+segmentation acrhictecture.
  • Figure 2: Crop predictions for DSTL image set in NIR domain. Light green represents a hit, dark green represents a miss, and red represents a false alarm.(a) Image. (b) Ground-truth mask. (c) DeepLabV3. (d) ViT (L$-$16). (e) LoraViT (L$-$16).
  • Figure 3: Tree predictions for RIT-18 image set in NIR domain. Light green represents a hit, dark green represents a miss, and red represents a false alarm. (a) Image. (b) Ground-truth mask. (c) DeepLabV3. (d) ViT (L$-$16). (e) LoraViT (L$-$16).