MutDet: Mutually Optimizing Pre-training for Remote Sensing Object Detection

TL;DR

MutDet tackles remote-sensing detection pre-training by addressing the feature-discrepancy between pre-trained backbone embeddings and detector features. It introduces a mutual enhancement module to deeply fuse object embeddings with encoder features, a contrastive alignment loss to co-optimize embeddings and predictions, and an auxiliary siamese head to bridge the pre-training and fine-tuning gap, all supported by SAM-generated pseudo-labels. Across DIOR-R, DOTA-v1.0, and OHD-SJTU, MutDet delivers state-of-the-art transfer with pronounced gains under limited data or training time, evidencing robust cross-domain pre-training for oriented remote-sensing object detection. This framework demonstrates effective utilization of pre-trained visual knowledge in dense, rotated-object scenes and provides a solid foundation for further improvements in detector–backbone synergy and proposal usage.

Abstract

Detection pre-training methods for the DETR series detector have been extensively studied in natural scenes, e.g., DETReg. However, the detection pre-training remains unexplored in remote sensing scenes. In existing pre-training methods, alignment between object embeddings extracted from a pre-trained backbone and detector features is significant. However, due to differences in feature extraction methods, a pronounced feature discrepancy still exists and hinders the pre-training performance. The remote sensing images with complex environments and more densely distributed objects exacerbate the discrepancy. In this work, we propose a novel Mutually optimizing pre-training framework for remote sensing object Detection, dubbed as MutDet. In MutDet, we propose a systemic solution against this challenge. Firstly, we propose a mutual enhancement module, which fuses the object embeddings and detector features bidirectionally in the last encoder layer, enhancing their information interaction.Secondly, contrastive alignment loss is employed to guide this alignment process softly and simultaneously enhances detector features' discriminativity. Finally, we design an auxiliary siamese head to mitigate the task gap arising from the introduction of enhancement module. Comprehensive experiments on various settings show new state-of-the-art transfer performance. The improvement is particularly pronounced when data quantity is limited. When using 10% of the DIOR-R data, MutDet improves DetReg by 6.1% in AP50. Codes and models are available at: https://github.com/floatingstarZ/MutDet.

Figures (4)

• Figure 1: Motivation of our method. (a) The predictive approaches detregup_detr utilize the embedding alignment task to learn visual knowledge from the pre-trained backbone. The feature discrepancy presoco between object embeddings and detector features impedes the effectiveness of pre-training. (b) Methods based on self-supervised learning aligndetpresoco circumvent feature discrepancy but can not sufficiently leverage the knowledge from pre-trained backbone. (c) Our approach employs contrastive alignment to achieve mutual learning between object embeddings and predictions, alleviating feature discrepancy. Simultaneously, we enhance the learning of visual knowledge by deeply fusing object embeddings with encoder features. (d) We use cosine similarity to measure the distance between object embeddings and predictions. The detector fails to fit the object embeddings, indicating the feature discrepancy problem.
• Figure 2: Overall architecture of the proposed MutDet. MutDet optimizes DETReg detreg and introduces SAM sam to generate proposals. It utilizes mutual enhancement module to cross-fuse the object embeddings and encoder features. Then, it uses contrastive alignment loss to optimize the enhanced object embeddings and predicted embeddings mutually. The auxiliary siamese head is proposed to alleviate the task gap between pre-training and fine-tuning, which shares parameters with the DETR decoder.
• Figure 3: A diagram illustrats the differences between the DETReg and MutDet with respect to supervisions. DETReg is only supervised by object embeddings via one pathway in the last decoder layer. In contrast, our MutDet is supervised through multiple pathways. The red line represents the supervision signal.
• Figure 4: Loss curves during the pre-training on DOTA-v1.0 dataset. The red curves are the losses when the mutual enhancement module is employed in pre-training, and the blue curves are the losses when not employed. Four losses are included: classification, regression, contrastive alignment, and angle prediction.