Few-Shot Object Detection with Sparse Context Transformers

TL;DR

The paper tackles few-shot object detection by addressing the difficulty of learning new categories with limited data. It introduces a Sparse Context Transformer (SCT) that plugs into an SSD detector and learns sparse affinities between prior boxes and contextual fields derived from multi-scale features, guided by a two-stage process: sparse relationship discovery and aggregation. Concretely, SCT computes a sparse affinity via $A = \text{softmax}(( \psi_{\gamma}(P) \psi_{\rho}(C)^{\top}) / \sqrt{C_s})$, derives $R = \text{sign}(A)$ to select relevant contextual fields, and then forms a context-aware prior representation $\hat{P} = P + \psi_{\xi}(W)$ with $W = \text{softmax}(R) \psi_{\eta}(C)$, producing final predictions $\hat{Y} = \text{softmax}(\hat{P} \Theta)$. The approach effectively leverages source-domain knowledge while automatically learning sparse, task-relevant context from few target-domain images, reducing class confusion and improving detection across PASCAL VOC and MS COCO benchmarks. Extensive ablations confirm the benefits of contextual fusion, sparsity, and the proposed Attention Focus module, and results show SCT achieving competitive or state-of-the-art performance in low-shot regimes and strong results on COCO 10/30-shot settings. The work offers a practical, plug-and-play enhancement for real-world detection under data scarcity and suggests future extensions to other backbones and benchmarks.

Abstract

Few-shot detection is a major task in pattern recognition which seeks to localize objects using models trained with few labeled data. One of the mainstream few-shot methods is transfer learning which consists in pretraining a detection model in a source domain prior to its fine-tuning in a target domain. However, it is challenging for fine-tuned models to effectively identify new classes in the target domain, particularly when the underlying labeled training data are scarce. In this paper, we devise a novel sparse context transformer (SCT) that effectively leverages object knowledge in the source domain, and automatically learns a sparse context from only few training images in the target domain. As a result, it combines different relevant clues in order to enhance the discrimination power of the learned detectors and reduce class confusion. We evaluate the proposed method on two challenging few-shot object detection benchmarks, and empirical results show that the proposed method obtains competitive performance compared to the related state-of-the-art.

Figures (3)

• Figure 1: Few-Shot Detection with Sparse-Context-Transformer. It consists of sparse relationship discovery and context aggregation, which can effectively utilize the context fields of few-shot tasks, improve the context awareness ability of each prior box, and solve the problem of object confusion in few-shot detection. The attention focus module can effectively help us learn task-related contextual fields.
• Figure 2: Visualization of the results with (w/) and without (w/o) Sparse Context Transformer (SCT) on the PASCAL VOC dataset. Different colored bounding boxes represent different categories.
• Figure 3: This figure shows comparison results on the MS COCO dataset. Baseline represents the work of Wang et al Yang_Wang_Chen_Liu_Qiao_2020. Different colored bounding boxes represent different categories.