Effective Guidance for Model Attention with Simple Yes-no Annotations

TL;DR

This work presents Crayon (Correcting Reasoning with Annotations of Yes Or No), offering effective, scalable, and practical solutions to rectify model attention using simple yes-no annotations.

Abstract

Modern deep learning models often make predictions by focusing on irrelevant areas, leading to biased performance and limited generalization. Existing methods aimed at rectifying model attention require explicit labels for irrelevant areas or complex pixel-wise ground truth attention maps. We present CRAYON (Correcting Reasoning with Annotations of Yes Or No), offering effective, scalable, and practical solutions to rectify model attention using simple yes-no annotations. CRAYON empowers classical and modern model interpretation techniques to identify and guide model reasoning: CRAYON-ATTENTION directs classic interpretations based on saliency maps to focus on relevant image regions, while CRAYON-PRUNING removes irrelevant neurons identified by modern concept-based methods to mitigate their influence. Through extensive experiments with both quantitative and human evaluation, we showcase CRAYON's effectiveness, scalability, and practicality in refining model attention. CRAYON achieves state-of-the-art performance, outperforming 12 methods across 3 benchmark datasets, surpassing approaches that require more complex annotations.

Figures (5)

• Figure 1: Left:Crayon uses purposefully simple yes-no annotations to guide a model to attend to relevant image areas (e.g., redirect from background to foreground), overcoming limitations of existing methods requiring complex annotations such as pixel-wise maps. Right:Crayon achieves state-of-the-art performance, surpassing 12 other methods across 3 benchmark datasets.
• Figure 2: Crayon-Pruning prunes the neurons activated by irrelevant concepts in the penultimate layer and fine-tunes the last layer. For example, in a smile classifier, Left: a neuron is activated by smiling mouth, which is relevant to smiling, while Right: another neuron is activated by irrelevant blond hair and is pruned.
• Figure 3: Example visualization shown to participants for attention annotations.
• Figure 4: For the Waterbirds dataset, even with annotations for just 10% of the training data points, both Crayon-Attention and Crayon-Pruning+Attention nearly reach their peak performance. Crayon-Pruning realizes its full effectiveness when annotations are provided for most neurons in the penultimate layer. The performance of Crayon-Pruning+Attention with no attention annotations differs from Crayon-Pruning's peak performance, as the latter retrains the entire model, while the former fine-tunes only the last layer. For each annotation number $n$, we run each method five times with different random seeds and report the average MGA and WGA values.
• Figure 5: For the Waterbirds dataset, both Crayon-Attention (left) and Crayon-Pruning+Attention (right) effectively correct model attention for wide range of $\alpha$ and $\beta$.