Lost in Context: The Influence of Context on Feature Attribution Methods for Object Recognition

TL;DR

This study investigates how context manipulation influences both model accuracy and feature attribution, providing insights into the reliance of object recognition models on contextual information as understood through the lens of feature attribution methods.

Abstract

Contextual information plays a critical role in object recognition models within computer vision, where changes in context can significantly affect accuracy, underscoring models' dependence on contextual cues. This study investigates how context manipulation influences both model accuracy and feature attribution, providing insights into the reliance of object recognition models on contextual information as understood through the lens of feature attribution methods. We employ a range of feature attribution techniques to decipher the reliance of deep neural networks on context in object recognition tasks. Using the ImageNet-9 and our curated ImageNet-CS datasets, we conduct experiments to evaluate the impact of contextual variations, analyzed through feature attribution methods. Our findings reveal several key insights: (a) Correctly classified images predominantly emphasize object volume attribution over context volume attribution. (b) The dependence on context remains relatively stable across different context modifications, irrespective of classification accuracy. (c) Context change exerts a more pronounced effect on model performance than Context perturbations. (d) Surprisingly, context attribution in `no-information' scenarios is non-trivial. Our research moves beyond traditional methods by assessing the implications of broad-level modifications on object recognition, either in the object or its context.

Figures (9)

• Figure 1: The predictions emphasize contextual information, while feature attribution for the top predicted class highlights the object in the image. Predictions were generated using a pre-trained ResNet50 model, and feature attribution was performed with GradCAM.
• Figure 2: Comparison of feature attribution methods derived from ResNet50 classifier for top-1 prediction.
• Figure 3: The top row provides images related to different varieties of our synthetic dataset named ImageNet-CS. The bottom row provides images showing variations of the ImageNet-9 dataset that we have considered for our experiments. We labelled ImageNet-9 images with its pre-trained ResNet50 classification—green, if corresponding with the original label; red, if not.
• Figure 4: Comparison of object and context volume attribution for ResNet50 and ResNet50-IN9L on the ImageNet-9 dataset using ScoreCAM. The figure shows higher context reliance in ResNet50-IN9L, highlighting the impact of training dataset size on model sensitivity to contextual cues.
• Figure 5: This plot illustrates the variation in volume attribution of Context for Correctly Classified Set and Wrongly Classified Set classifications, using ResNet50 as our backbone architecture. The feature attributions were generated based on the model's top prediction. The volume attributions for both ImageNet-9 and ImageNet-CS datasets are presented.