OpenSlot: Mixed Open-Set Recognition with Object-Centric Learning

TL;DR

This work defines mixed open-set recognition (OSR), where negatives can contain both known and unknown classes, creating a super-label shift beyond traditional full-label OSR. It introduces OpenSlot, an object-centric, slot-based framework augmented with an anti-noise-slot (ANS) mechanism to separate noise slots from true class semantics, enabling both robust OSR scoring and open-set object localization without bounding boxes. Through unsupervised slot pretraining and a two-classifier training scheme with Hungarian matching, OpenSlot achieves state-of-the-art results on conventional OSR benchmarks and substantially improves performance on mixed OSR tasks, while remaining computationally efficient. The approach also demonstrates competitive open-set object detection by leveraging attention masks as proposals, highlighting strong generalization and practical applicability in real-world open-world vision tasks.

Abstract

Existing open-set recognition (OSR) studies typically assume that each image contains only one class label, with the unknown test set (negative) having a disjoint label space from the known test set (positive), a scenario referred to as full-label shift. This paper introduces the mixed OSR problem, where test images contain multiple class semantics, with both known and unknown classes co-occurring in the negatives, leading to a more complex super-label shift that better reflects real-world scenarios. To tackle this challenge, we propose the OpenSlot framework, based on object-centric learning, which uses slot features to represent diverse class semantics and generate class predictions. The proposed anti-noise slot (ANS) technique helps mitigate the impact of noise (invalid or background) slots during classification training, addressing the semantic misalignment between class predictions and ground truth. We evaluate OpenSlot on both mixed and conventional OSR benchmarks. Without elaborate designs, our method not only excels existing approaches in detecting super-label shifts across OSR tasks, but also achieves state-of-the-art performance on conventional benchmarks. Meanwhile, OpenSlot can localize class objects without using bounding boxes during training, demonstrating competitive performance in open-set object detection and potential for generalization.

Figures (9)

• Figure 1: (a) Conventional OSR assumes test samples only contain one class semantic (single-label) and regards images from new classes as unknown. (b) We propose the mixed OSR problem, which tests on images with multiple classes. The negative samples contain known and unknown classes and require the classifier to accurately detect the occurrence of unknown classes. (3) Existing methods can easily ignore the unknown class due to the existence of the known content. To address this issue, we propose using slots for semantic representation and producing independent class predictions. Thus, we learn a clear known/unknown decision boundary. Besides, our method can localize the unknown class and realize open-set object detection (OSOD).
• Figure 2: OpenSlot overview. In (1), we pre-train Dinosaur dinosaur (following the encoder-decoder design, the encoder can be either ResNet or Transformer architecture) on the known training images, using slots $S$ to learn class semantics. Next (2), we add two classifiers, the noise classifier $\varphi_{nz}$ and the main classifier $\varphi_{fg}$, trained with $L_{nz}$ (Eq. \ref{['eq:nz_loss']}) and $L_{match}$ (Eq. \ref{['eq:match_loss']}) respectively, to identify the noise (invalid and background) slots and implement semantic classification with the proposed ANS technique (Fig. \ref{['fig:ANS_presentation']}). During inference (3), we score slot-based class predictions to address the mixed OSR.
• Figure 3: Anti-noise-slot (ANS). After pre-training, we learned slot $S$ to (a) represent the input's diverse semantics (shown in different colors) and get their attention masks (each slot has an attention mask). (b) In each step of the classification training, we: 1) first merge the null slot mask with the invalid slot mask $M_{inv}$ (Eq. \ref{['eq:invalid']}) to get the pseudo label $M_{nz}$; (2) use $M_{nz}$ to train the noise classifier $\varphi_{nz}$; 3) get the mask $M$ (Eq. \ref{['eq:nosiy_mask']}) of high-confidence noise prediction from $\varphi_{nz}$ to calibrate (Eq. \ref{['eq:match_loss']}) the assignment cost of Hungarian matching, enabling $\varphi_{fg}$ to focus on true class semantics.
• Figure 4: To compare OpenSlot with the Pure Slot baseline, we divide all slots into the foreground (FG) and Noise groups based on the semantics denoted by each slot. We then show the distribution of the max-logit slot (with the highest logit value from $\varphi_{fg}$) across these two groups (Left). Besides, we present each group's logit norm (right) (after Min-max normalization).
• Figure 5: Display of the semantic misalignment problem. After pre-training, we learn slots for semantic representation (in different colors). In Pure Slot, we train the main classifier $\varphi_{fg}$ with vanilla Hungarian matching. When sorting (ascending) all slots by $\varphi_{fg}$'s output and displaying their attention masks (Top 6 $\rightarrow$ Top 1), we find that the noise (invalid & background) slots have larger logit values than the true class slots (marked with red), indicating that $\varphi_{fg}$ is confused by noise slots.