Learning De-Biased Representations for Remote-Sensing Imagery

TL;DR

debLoRA is a generic training approach that works with any LoRA variants to yield debiased features that can diversify minor class features based on the shared attributes with major classes, where the attributes are obtained by a simple step of clustering.

Abstract

Remote sensing (RS) imagery, requiring specialized satellites to collect and being difficult to annotate, suffers from data scarcity and class imbalance in certain spectrums. Due to data scarcity, training any large-scale RS models from scratch is unrealistic, and the alternative is to transfer pre-trained models by fine-tuning or a more data-efficient method LoRA. Due to class imbalance, transferred models exhibit strong bias, where features of the major class dominate over those of the minor class. In this paper, we propose debLoRA, a generic training approach that works with any LoRA variants to yield debiased features. It is an unsupervised learning approach that can diversify minor class features based on the shared attributes with major classes, where the attributes are obtained by a simple step of clustering. To evaluate it, we conduct extensive experiments in two transfer learning scenarios in the RS domain: from natural to optical RS images, and from optical RS to multi-spectrum RS images. We perform object classification and oriented object detection tasks on the optical RS dataset DOTA and the SAR dataset FUSRS. Results show that our debLoRA consistently surpasses prior arts across these RS adaptation settings, yielding up to 3.3 and 4.7 percentage points gains on the tail classes for natural to optical RS and optical RS to multi-spectrum RS adaptations, respectively, while preserving the performance on head classes, substantiating its efficacy and adaptability.

Figures (4)

• Figure 1: Long-tailed Problems. This figure shows 1) ORS datasets (take DOTA xia2018dota as an example) have the long-tailed distribution issue. 2) Model adaptation methods suffer from weak performance in tail classes.
• Figure 2: Two key steps of debLoRA: feature clustering and calibration. (a) The baseline LoRA feature space is biased towards head classes. Red crosses represent head class samples, and blue triangles represent tail class samples. The blue star indicates the center of tail class samples. Dashed blue triangles show the validation samples of the tail class wrongly embedded in the head class region, indicating the model bias towards head classes. (b) We cluster all features (clusters denoted by gray dotted boundaries) regardless of class labels. $A$, $B$ and $C$ are cluster centers used to generate a de-biased center $D$, as in Eq. 2. (c) We calibrate the tail class features by "moving" them closer to $D$, as in Eq. 3. After these steps, we train the debLoRA module on the calibrated features of tail classes (together with the original head class features).
• Figure 3: t-SNE visualization of validation samples and clusters. The first column shows the distribution of helicopter (tail) and ship (head) validation samples. Subfigures (c)-(g) are the clusters and their centers when $K$=5 in $K$-means. In (h), the dotted lines and stars indicate that we compute a de-biased center for the tail class (helicopter) by weighted averaging the five cluster centers, and the blue star is the original biased center of helicopter training samples.
• Figure A0: Constraints of the SAR datasets' test sets. This figure illustrates the per-category test sample distribution of (a) the FUSAR dataset and (b) the SRSD dataset. The FUSAR dataset suffers from insufficient test samples and vaguely defined classes (indicated by "$\ast$"). Similarly, the SRSDD dataset also has the issue of insufficient test samples.