Low-Rank Adaption on Transformer-based Oriented Object Detector for Satellite Onboard Processing of Remote Sensing Images

TL;DR

This work tackles bandwidth-constrained satellite uplinks by proposing LoRA-Det, a parameter-efficient fine-tuning framework that distributes Low Rank Adaptation (LoRA) across a transformer-based backbone and the detection head for oriented object detection in remote sensing. It combines LoRA updates with selective full fine-tuning for convolutional modules, and uses SVD-based low-rank approximation to choose per-module ranks, achieving substantial parameter reduction while preserving near-full fine-tuning performance. Empirical results on DOTAv1.0, HRSC2016, and DIOR-R show LoRA-Det can update about $12.4\%$ of parameters and reach roughly $98\%$ of the full fine-tuning performance, outperforming lightweight detectors under similar parameter budgets. These findings demonstrate a practical pathway for robust, bandwidth-efficient onboard processing of high-resolution satellite imagery with strong generalization capabilities.

Abstract

Deep learning models in satellite onboard enable real-time interpretation of remote sensing images, reducing the need for data transmission to the ground and conserving communication resources. As satellite numbers and observation frequencies increase, the demand for satellite onboard real-time image interpretation grows, highlighting the expanding importance and development of this technology. However, updating the extensive parameters of models deployed on the satellites for spaceborne object detection model is challenging due to the limitations of uplink bandwidth in wireless satellite communications. To address this issue, this paper proposes a method based on parameter-efficient fine-tuning technology with low-rank adaptation (LoRA) module. It involves training low-rank matrix parameters and integrating them with the original model's weight matrix through multiplication and summation, thereby fine-tuning the model parameters to adapt to new data distributions with minimal weight updates. The proposed method combines parameter-efficient fine-tuning with full fine-tuning in the parameter update strategy of the oriented object detection algorithm architecture. This strategy enables model performance improvements close to full fine-tuning effects with minimal parameter updates. In addition, low rank approximation is conducted to pick an optimal rank value for LoRA matrices. Extensive experiments verify the effectiveness of the proposed method. By fine-tuning and updating only 12.4$\%$ of the model's total parameters, it is able to achieve 97$\%$ to 100$\%$ of the performance of full fine-tuning models. Additionally, the reduced number of trainable parameters accelerates model training iterations and enhances the generalization and robustness of the oriented object detection model. The source code is available at: \url{https://github.com/fudanxu/LoRA-Det}.

Figures (10)

• Figure 1: The transmission limitations in the uplink bandwidth of spaceborne object detection models.
• Figure 2: The structure of LoRA module.
• Figure 3: The architecture of LoRA module in Swin Transformer block.
• Figure 4: The architecture of the proposed LoRA-Det.
• Figure 5: Parameter distribution of oriented object detection model Oriented RCNN.