Scalpel-SAM: A Semi-Supervised Paradigm for Adapting SAM to Infrared Small Object Detection

TL;DR

Scalpel-SAM addresses infrared small object detection under data scarcity by retooling SAM with a Hierarchical MoE Adapter that encodes physics-based priors through four white-box operators. The two-stage paradigm first distills SAM into an expert teacher using limited labeled data, then transfers knowledge to a lightweight downstream model via pseudo-labels, achieving near- or superior performance to fully supervised baselines with only 10% of labels. Extensive experiments across four IR-SOT datasets demonstrate strong data efficiency, and ablations justify the necessity of dynamic routing, white-box design, and adapter placement. This domain-principle-driven PEFT framework offers a practical path to adapting foundation models to other specialized, data-scarce domains such as medical imaging.

Abstract

Infrared small object detection urgently requires semi-supervised paradigms due to the high cost of annotation. However, existing methods like SAM face significant challenges of domain gaps, inability of encoding physical priors, and inherent architectural complexity. To address this, we designed a Hierarchical MoE Adapter consisting of four white-box neural operators. Building upon this core component, we propose a two-stage paradigm for knowledge distillation and transfer: (1) Prior-Guided Knowledge Distillation, where we use our MoE adapter and 10% of available fully supervised data to distill SAM into an expert teacher (Scalpel-SAM); and (2) Deployment-Oriented Knowledge Transfer, where we use Scalpel-SAM to generate pseudo labels for training lightweight and efficient downstream models. Experiments demonstrate that with minimal annotations, our paradigm enables downstream models to achieve performance comparable to, or even surpassing, their fully supervised counterparts. To our knowledge, this is the first semi-supervised paradigm that systematically addresses the data scarcity issue in IR-SOT using SAM as the teacher model.

Figures (3)

• Figure 1: The proposed paradigm and the visual evidence of the challenge. (a) Our two-stage paradigm. Stage One refines SAM (robot icon) into an expert teacher via our MoE adapter; Stage Two trains a lightweight student model (white robot) using teacher-generated pseudo-labels. (b) Visual comparison on the SIRST3 dataset. From top to bottom, the rows display: the original image, the generic SAM's segmentation, our Scalpel-SAM's precise result, and the GT.
• Figure 2: The architecture of our two-stage paradigm. Stage 1 (Top-Right): Our trainable Hierarchical MoE Adapter (containing a router and four expert operators: PIMDO, SPD, HPLSM, TGDS) is injected into the frozen SAM ViT Encoder. Stage 2 (Bottom-Right): The resulting expert teacher generates pseudo labels from unlabeled data to supervise a lightweight downstream student.
• Figure 3: Qualitative comparison on the IRSTD-1K dataset. Each row from top to bottom displays: the original infrared image, the 3D brightness map of the original image, the results of PAL, the results of the fully supervised method, the results of our method, and the ground truth (GT). For the predictions in this figure, red represents true positives (TP), orange represents false positives (FP), and green represents false negatives (FN). The visual results show that our method significantly outperforms PAL in terms of performance and achieves results that are close to, or even surpass, those of the fully supervised method.