Informed Mixing -- Improving Open Set Recognition via Attribution-based Augmentation

TL;DR

Open set recognition requires detecting unseen classes during inference, a challenge for modern vision models. The paper introduces GradMix, an attribution-based augmentation that masks highly activated regions using LayerCAM-derived maps and patches another image within the batch to force learning from broader regions; it combines supervised and self-supervised contrastive learning and aggregates at multiple layers. A distance-based OSR detector is used, and GradMix yields state-of-the-art results across OSR, close-set classification, and OOD tasks, while also improving robustness to common corruptions and boosting SSL linear probe performance. The approach demonstrates that encouraging feature diversity via gradient-informed augmentation enhances generalization across diverse data regimes and backbone architectures. Overall, GradMix offers a practical, versatile strategy to improve open-world recognition and model robustness.

Abstract

Open set recognition (OSR) is devised to address the problem of detecting novel classes during model inference. Even in recent vision models, this remains an open issue which is receiving increasing attention. Thereby, a crucial challenge is to learn features that are relevant for unseen categories from given data, for which these features might not be discriminative. To facilitate this process and "optimize to learn" more diverse features, we propose GradMix, a data augmentation method that dynamically leverages gradient-based attribution maps of the model during training to mask out already learned concepts. Thus GradMix encourages the model to learn a more complete set of representative features from the same data source. Extensive experiments on open set recognition, close set classification, and out-of-distribution detection reveal that our method can often outperform the state-of-the-art. GradMix can further increase model robustness to corruptions as well as downstream classification performance for self-supervised learning, indicating its benefit for model generalization.

Figures (11)

• Figure 1: An illustration of GradMix’s effectiveness using attribution maps (green and red boxes for in set (car, boat) and open set (airplane, bird) samples respectively; dashed-line boxes indicate models without GradMix, while solid-line boxes represent models with GradMix). GradMix enables models to focus on broader areas in data for in set samples while capturing object regions more effectively for open set samples.
• Figure 2: Graphical illustration of GradMix. Three blocks with dashed borderlines refer to the procedures: A. Data is fed into the feature extractor during forward propagation; B: Attribution maps are computed using the internal feature maps and LayerCAM method. The most activated area is selected (highlighted using a white star in the graph); C: A random sample from the same minibatch is resized and patched on the most activated area.
• Figure 3: Graphical illustration of the proposed layer aggregation for GradMix: the attribution maps calculated using individual layers are summed to get $\mathbf{M}_\textit{layercam}$.
• Figure 4: Left: OSR performances of the models with different augmentation methods on CIFAR10 and TinyImageNet protocols. Clear improvements can be brought by extra data augmentations. And GradMix performs best among all augmentation methods. Right: OSR performances of models with GradMix computed using different layers and layer aggregation in ResNet18. The results indicate that different layers can produce non-negligible changes in performance. And layer aggregation can in general provide better performance than single layers.
• Figure 5: Classification accuracy drop of the models trained with and without GradMix on CIFAR10 and TinyImgNet. The average accuracy drop $\bar{D}_{c}$s are lower for the models with GradMix for most corruption types on both datasets.