Balanced Hierarchical Contrastive Learning with Decoupled Queries for Fine-grained Object Detection in Remote Sensing Images

TL;DR

This work tackles fine-grained object detection in remote sensing where hierarchical labels are available but learning is challenged by data imbalance and the potential conflict between semantic grouping and localization. It introduces a Balanced Hierarchical Contrastive Loss (BHCL) with learnable class prototypes and a decoupled learning strategy inside DETR to balance learning across hierarchy levels while separating classification from localization. BHCL embeds hierarchical semantics through multi-level contrastive learning and prototype balancing, and the decoupled queries prevent interference with bounding-box regression. Experiments on ShipRSImageNet and FAIR1M datasets show state-of-the-art gains in fine-grained accuracy, with comprehensive ablations validating the contribution of each component and analyses highlighting the impact of data imbalance.

Abstract

Fine-grained remote sensing datasets often use hierarchical label structures to differentiate objects in a coarse-to-fine manner, with each object annotated across multiple levels. However, embedding this semantic hierarchy into the representation learning space to improve fine-grained detection performance remains challenging. Previous studies have applied supervised contrastive learning at different hierarchical levels to group objects under the same parent class while distinguishing sibling subcategories. Nevertheless, they overlook two critical issues: (1) imbalanced data distribution across the label hierarchy causes high-frequency classes to dominate the learning process, and (2) learning semantic relationships among categories interferes with class-agnostic localization. To address these issues, we propose a balanced hierarchical contrastive loss combined with a decoupled learning strategy within the detection transformer (DETR) framework. The proposed loss introduces learnable class prototypes and equilibrates gradients contributed by different classes at each hierarchical level, ensuring that each hierarchical class contributes equally to the loss computation in every mini-batch. The decoupled strategy separates DETR's object queries into classification and localization sets, enabling task-specific feature extraction and optimization. Experiments on three fine-grained datasets with hierarchical annotations demonstrate that our method outperforms state-of-the-art approaches.

Figures (9)

• Figure 1: Illustration of the hierarchical label structure in remote sensing datasets. Categories are organized in a coarse-to-fine manner, with leaf nodes representing fine-grained categories. Embedding this structure into the representation learning process poses two primary challenges: (a) data imbalances across and within hierarchical levels, and (b) the conflict between semantically grouping objects under the same parent class while preserving their distinct spatial locations. Additionally, this structure enables flexible reassignment of ambiguous objects to appropriate parent categories.
• Figure 2: Architecture of the proposed method. A backbone network and an encoder are employed to extract and refine image features, respectively. These refined features initialize a set of learnable object queries, which are then decoupled into classification and localization queries. At each decoder layer, these queries first interact through a shared self-attention module, followed by two parallel streams for classification and localization tasks. Each task-specific stream contains a cross-attention module and a feed-forward network. Finally, dedicated branches in the prediction head generate class predictions from the refined classification queries and regress bounding boxes from the localization queries. During training, bipartite matching assigns ground truth to the predictions, and the $L_1$ loss, IoU loss, classification loss, and the proposed balanced hierarchical contrastive loss (BHCL) are computed for model optimization. BHCL embeds the class hierarchy into the object representation space within the DETR framework, introducing learnable prototypes for each class in the hierarchy to ensure balanced contributions from all classes to the loss computation in every mini-batch.
• Figure 3: t-SNE Visualization of Classification Queries: (a) OrientedFormer at Level 2, (b) Proposed Method at Level 2, (c) OrientedFormer at Level 3, (d) Proposed Method at Level 3. The full names of the abbreviated subcategories under the coarse-grained categories "Warship" and "Merchant" are provided in the supplementary material.
• Figure 4: Comparison of Different Weights for Balanced Hierarchical Contrastive Loss.
• Figure 5: Comparison of Detection Results on ShipRSImageNet (top) and FAIR1M-v2.0 (bottom) Validation Sets: (a) Ground Truth, (b) OrientedFormer, (c) Proposed Method (Best Results).