Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection

Yuxuan Li, Xiang Li, Yunheng Li, Yicheng Zhang, Yimian Dai, Qibin Hou, Ming-Ming Cheng, Jian Yang

TL;DR

The paper tackles the challenge of detecting objects across diverse remote-sensing modalities and tasks with a unified approach. It defines the Multi-Modal Datasets and Multi-Task Object Detection (M2Det) task and introduces SM3Det, featuring a grid-level sparse MoE backbone and dynamic submodule optimization to align learning across modalities. By merging SARDet-100K, DOTA, and DroneVehicle into the SOI-Det benchmark, SM3Det demonstrates performance gains over specialized models and prior SOTA methods, while maintaining parameter efficiency and flexibility across backbones and detectors. The work has practical implications for edge devices and multi-sensor ecosystems, with potential extensions to other domains such as medical imaging and autonomous systems.

Abstract

With the rapid advancement of remote sensing technology, high-resolution multi-modal imagery is now more widely accessible. Conventional Object detection models are trained on a single dataset, often restricted to a specific imaging modality and annotation format. However, such an approach overlooks the valuable shared knowledge across multi-modalities and limits the model's applicability in more versatile scenarios. This paper introduces a new task called Multi-Modal Datasets and Multi-Task Object Detection (M2Det) for remote sensing, designed to accurately detect horizontal or oriented objects from any sensor modality. This task poses challenges due to 1) the trade-offs involved in managing multi-modal modelling and 2) the complexities of multi-task optimization. To address these, we establish a benchmark dataset and propose a unified model, SM3Det (Single Model for Multi-Modal datasets and Multi-Task object Detection). SM3Det leverages a grid-level sparse MoE backbone to enable joint knowledge learning while preserving distinct feature representations for different modalities. Furthermore, it integrates a consistency and synchronization optimization strategy using dynamic learning rate adjustment, allowing it to effectively handle varying levels of learning difficulty across modalities and tasks. Extensive experiments demonstrate SM3Det's effectiveness and generalizability, consistently outperforming specialized models on individual datasets. The code is available at https://github.com/zcablii/SM3Det.

SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection

TL;DR

The paper tackles the challenge of detecting objects across diverse remote-sensing modalities and tasks with a unified approach. It defines the Multi-Modal Datasets and Multi-Task Object Detection (M2Det) task and introduces SM3Det, featuring a grid-level sparse MoE backbone and dynamic submodule optimization to align learning across modalities. By merging SARDet-100K, DOTA, and DroneVehicle into the SOI-Det benchmark, SM3Det demonstrates performance gains over specialized models and prior SOTA methods, while maintaining parameter efficiency and flexibility across backbones and detectors. The work has practical implications for edge devices and multi-sensor ecosystems, with potential extensions to other domains such as medical imaging and autonomous systems.

Abstract

With the rapid advancement of remote sensing technology, high-resolution multi-modal imagery is now more widely accessible. Conventional Object detection models are trained on a single dataset, often restricted to a specific imaging modality and annotation format. However, such an approach overlooks the valuable shared knowledge across multi-modalities and limits the model's applicability in more versatile scenarios. This paper introduces a new task called Multi-Modal Datasets and Multi-Task Object Detection (M2Det) for remote sensing, designed to accurately detect horizontal or oriented objects from any sensor modality. This task poses challenges due to 1) the trade-offs involved in managing multi-modal modelling and 2) the complexities of multi-task optimization. To address these, we establish a benchmark dataset and propose a unified model, SM3Det (Single Model for Multi-Modal datasets and Multi-Task object Detection). SM3Det leverages a grid-level sparse MoE backbone to enable joint knowledge learning while preserving distinct feature representations for different modalities. Furthermore, it integrates a consistency and synchronization optimization strategy using dynamic learning rate adjustment, allowing it to effectively handle varying levels of learning difficulty across modalities and tasks. Extensive experiments demonstrate SM3Det's effectiveness and generalizability, consistently outperforming specialized models on individual datasets. The code is available at https://github.com/zcablii/SM3Det.
Paper Structure (16 sections, 7 equations, 10 figures, 12 tables)

This paper contains 16 sections, 7 equations, 10 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Dataset Object Detection
  4. Multi-Task Learning
  5. Multi-Task Learning
  6. Mixture of Experts (MoE)
  7. Methods
  8. Task Definition
  9. Methodological Overview
  10. Grid-level MoE
  11. Dynamic Submodule Optimization (DSO)
  12. Experiments and Analysis
  13. Main Results
  14. Ablation Study and Analysis
  15. Limitation and Future Work
  16. ...and 1 more sections

Figures (10)

  • Figure 1: Comparison of tasks: (a) Spatially Aligned Multi-Modality, (b) Traditional Single Dataset, and (c) M2Det. M2Det aims to utilize a unified model for detecting objects in any modality, handling various detection tasks.
  • Figure 2: A conceptual illustration of SM3Det model. "HBB": horizontal bounding box, "OBB": oriented bounding box.
  • Figure 3: Reweighting curves for various temperature ($\tau$).
  • Figure 4: SM3Det on different backbones.
  • Figure 5: SM3Det on different detector heads.
  • ...and 5 more figures