BackMix: Regularizing Open Set Recognition by Removing Underlying Fore-Background Priors

TL;DR

This paper tackles open set recognition by identifying fore-background priors as a key susceptibility. It proposes BackMix, a CAM-guided, cut-and-paste augmentation that decorrelates foreground objects from background contexts by mixing a target image with a background from another image, using a fixed cut and a CAM-based foreground mask. The authors provide a theoretical framing via a mutual-information decomposition and validate the method with extensive experiments, showing consistent improvements over both standard OSR baselines and recent state-of-the-art methods across unknown detection, open set classification, and OOD detection, while also reducing reliance on auxiliary outliers. The approach is simple to implement, does not affect inference, and complements existing frameworks, suggesting practical impact for robust recognition in open-world settings.

Abstract

Open set recognition (OSR) requires models to classify known samples while detecting unknown samples for real-world applications. Existing studies show impressive progress using unknown samples from auxiliary datasets to regularize OSR models, but they have proved to be sensitive to selecting such known outliers. In this paper, we discuss the aforementioned problem from a new perspective: Can we regularize OSR models without elaborately selecting auxiliary known outliers? We first empirically and theoretically explore the role of foregrounds and backgrounds in open set recognition and disclose that: 1) backgrounds that correlate with foregrounds would mislead the model and cause failures when encounters 'partially' known images; 2) Backgrounds unrelated to foregrounds can serve as auxiliary known outliers and provide regularization via global average pooling. Based on the above insights, we propose a new method, Background Mix (BackMix), that mixes the foreground of an image with different backgrounds to remove the underlying fore-background priors. Specifically, BackMix first estimates the foreground with class activation maps (CAMs), then randomly replaces image patches with backgrounds from other images to obtain mixed images for training. With backgrounds de-correlated from foregrounds, the open set recognition performance is significantly improved. The proposed method is quite simple to implement, requires no extra operation for inferences, and can be seamlessly integrated into almost all of the existing frameworks. The code is released on https://github.com/Vanixxz/BackMix.

Figures (11)

• Figure 1: Illustration of operations applied on the generated dataset. Raw, FG, and BG represent the original image, the foreground of the image, and the background of the image, respectively. The star (*) on Raw and BG denotes that another source image is randomly sampled from the dataset.
• Figure 2: Illustration of three different models compared in the OE experiment. (a) The traditional OE method trains known samples and outliers separately. (b) CatImg concatenates outliers to known samples, serving as constructed image backgrounds. (c) FtAvg inputs known samples and outliers to the backbone separately and restricts them from interacting under simulated GAP.
• Figure 3: Illustration of BackMix. BackMix first estimates and masks the foreground of the background image, then randomly cuts patches and pastes them on the target image to obtain the mixed image as the training sample.
• Figure 4: Parameter Analysis for BackMix, where (a) presents the closed-set classification accuracy, (b) presents the AUROC with varying cut size $s$ and mask ratio $k$. The values of open set metric AUROC are averaged on the six unknown datasets.
• Figure 5: Examples of the estimated foreground masks, and labels have been annotated below the corresponding image. The rough segmentation using CAM can effectively estimate the foreground.

Theorems & Definitions (4)

• Theorem 1
• Theorem 2
• proof
• proof