Bringing the Context Back into Object Recognition, Robustly

TL;DR

The paper addresses the vulnerability of object recognition to background cues by proposing Localize to Recognize Robustly (L2R2), which localizes the foreground before recognition to enable both robust FG-only and context-aware full predictions. It decomposes images into FG and BG using zero-shot detectors, trains independent FG and FULL classifiers, and fuses their outputs with an interpretable module, achieving improved robustness to BG distribution shifts. The approach yields significant gains in supervised and zero-shot settings across diverse datasets, including Hard ImageNet and FungiTastic, and demonstrates notable improvements for vision-language models like SigLIP2 and BioCLIP. While effective, the method relies on zero-shot detectors and incurs extra computation from training two classifiers, with future work aiming to extend to detection tasks and enhance efficiency.

Abstract

In object recognition, both the subject of interest (referred to as foreground, FG, for simplicity) and its surrounding context (background, BG) may play an important role. However, standard supervised learning often leads to unintended over-reliance on the BG, limiting model robustness in real-world deployment settings. The problem is mainly addressed by suppressing the BG, sacrificing context information for improved generalization. We propose "Localize to Recognize Robustly" (L2R2), a novel recognition approach which exploits the benefits of context-aware classification while maintaining robustness to distribution shifts. L2R2 leverages advances in zero-shot detection to localize the FG before recognition. It improves the performance of both standard recognition with supervised training, as well as multimodal zero-shot recognition with VLMs, while being robust to long-tail BGs and distribution shifts. The results confirm localization before recognition is possible for a wide range of datasets and they highlight the limits of object detection on others

Figures (12)

• Figure 1: The complementarity of foreground ( fg) and background ( bg) in recognition. The standard approach, background suppression, makes correct identification in (a) nearly impossible, and difficult in (b); the spectacled bear is the most herbivorous of all bear species. On the other hand, rare backgrounds with possibly huge diversity hurt classification -- (d) shows a cheetah after a snowfall in South Africa, not a snow leopard. In generated content (e), any fg can appear on any bg as in ChatGPT 4o's response to "a dolphin on the moon".
• Figure 2: VLM (CLIP-B) -- zero-shot recognition with ground truth prompts and selected distractors. In the top example, recognition fails on the foreground (left, crop of a tight object bounding box). In the bottom, it fails on the full image (right). The proposed L2R$^2$ fusion is correct both times.
• Figure 3: The "Localize to Recognize Robustly" approach to context-aware recognition -- L2R$^2$ -- proceeds in three stages: (1) decomposition of image $x$ into fg and bg by zero-shot detection, possibly exploiting the predictions of full for prompt generation (2) independent modelling of the fg and the context-aware full (original image), which also serves as a fallback option when detection fails, and (3) fusion that robustly combines the representations from stage (2) to form the output prediction $p(k|x)$.
• Figure 4: The unexpected role of shape in bg modelling. When investigating the results on the Hard ImageNet dataset, many examples where found where full image prediction is incorrect but bothfg and bg (with shape) predictions are correct Possible explanation: the mask provides the bg model with information about the location of the target object and its shape, information not available to unlike the full image model.
• Figure 5: Examples where fg model is correct and both full image and bg models are incorrect on Hard ImageNet - Long Tail.