MOS: Modeling Object-Scene Associations in Generalized Category Discovery

TL;DR

This work tackles Generalized Category Discovery (GCD) by reframing scene context from noise to a useful prior, addressing the Ambiguity Challenge that perturbs object-scene interpretation. It introduces Modeling Object-Scene Associations (MOS), a dual-branch framework with a lightweight, MLP-based scene-awareness module and a teacher-stabilized scene feature pathway, coupled with zero-shot saliency segmentation to extract object regions. MOS integrates both contrastive and classification losses across labeled and unlabeled data, uses a Hungarian matcher for final predictions, and demonstrates strong gains on fine-grained GCD benchmarks—achieving an exceptional 4% average accuracy improvement and up to 9% gains over state-of-the-art on Semantic Shift Benchmark subsets. The approach emphasizes the practical value of scene information in GCD, shows robust ablations and analyses, and provides publicly accessible code, highlighting its potential to improve real-world recognition when scene cues are leveraged rather than discarded.

Abstract

Generalized Category Discovery (GCD) is a classification task that aims to classify both base and novel classes in unlabeled images, using knowledge from a labeled dataset. In GCD, previous research overlooks scene information or treats it as noise, reducing its impact during model training. However, in this paper, we argue that scene information should be viewed as a strong prior for inferring novel classes. We attribute the misinterpretation of scene information to a key factor: the Ambiguity Challenge inherent in GCD. Specifically, novel objects in base scenes might be wrongly classified into base categories, while base objects in novel scenes might be mistakenly recognized as novel categories. Once the ambiguity challenge is addressed, scene information can reach its full potential, significantly enhancing the performance of GCD models. To more effectively leverage scene information, we propose the Modeling Object-Scene Associations (MOS) framework, which utilizes a simple MLP-based scene-awareness module to enhance GCD performance. It achieves an exceptional average accuracy improvement of 4% on the challenging fine-grained datasets compared to state-of-the-art methods, emphasizing its superior performance in fine-grained GCD. The code is publicly available at https://github.com/JethroPeng/MOS

Figures (8)

• Figure 1: Illustration of the ambiguity challenge. The top row illustrates the potential for base objects in novel scenes to be perceived as novel categories. The bottom row shows misclassification risks when novel objects are placed in base scenes. Ambiguity challenge is the primary factor that leads to the misinterpretation of scene information.
• Figure 2: Comparison of the performance on the 4 subsets of the CUB dataset. The red box highlights scenarios involving two types of ambiguities. We compare the performance variations between the scene-removed image (green) and the original image (dark blue) as inputs. Our analysis reveals that the most significant performance improvement occurs in situations involving a novel-base relationship conflict between object and scene. In other situations, there is a slight decline in performance. Furthermore, employing dual-branch network for training (yellow) leads to a notable performance enhancement across four subsets.
• Figure 3: Modeling Object-Scene Associations (MOS) Framework. The framework adopts a dual-branch design: one branch processes the original image, and the other handles the segmented object image. Both branches share two core components: the Backbone$f$ and the Scene-awareness Module$\theta$. During training, the original image is segmented using a universal saliency segmentation model to extract the object. The scene regions of the object image are then filled with the mean pixel value. Both the original and object images are fed into the Backbone. After extracting features $v_o$ from the object image and $v_x$ from the original image, the scene features $v_s$ are obtained from $v_x$. In the scene-awareness module, $v_x$ and $v_s$, as well as $v_o$ and $v_s$, interact to produce the classification output and compute the losses. During evaluation, only the output from the object branch is used.
• Figure 4: Display of CUB Scene Information. The left side shows category counts with a log-transformed y-axis to better illustrate the distribution. The right side shows category examples, illustrating the dataset's diversity.
• Figure 5: Ablation Study on Parameters (left) and Shared Weights (right) on the CUB dataset. The ablation study on parameters shows that the method is robust to parameter changes. The ablation study on shared weights reveals that shared weights primarily influence the inference of novel classes