TdAttenMix: Top-Down Attention Guided Mixup

TL;DR

TdAttenMix introduces a top-down attention guided mixup that combines a Top-down Attention Guided Module with bottom-up saliency to crop task-relevant patches and a Area-Attention Label Mixing scheme to adjust label proportions based on both region area and attention. The method, applicable to ResNet and ViT, yields state-of-the-art or competitive top-1 accuracy across eight benchmarks, improves downstream segmentation tasks, and enhances robustness to occlusion and out-of-distribution data. A gaze-driven inconsistency metric demonstrates that TdAttenMix reduces image-label misalignment compared to purely bottom-up approaches. Overall, the approach highlights the practical value of task-guided attention in data augmentation and offers a quantitative framework for studying image-label consistency in mixup.

Abstract

CutMix is a data augmentation strategy that cuts and pastes image patches to mixup training data. Existing methods pick either random or salient areas which are often inconsistent to labels, thus misguiding the training model. By our knowledge, we integrate human gaze to guide cutmix for the first time. Since human attention is driven by both high-level recognition and low-level clues, we propose a controllable Top-down Attention Guided Module to obtain a general artificial attention which balances top-down and bottom-up attention. The proposed TdATttenMix then picks the patches and adjust the label mixing ratio that focuses on regions relevant to the current label. Experimental results demonstrate that our TdAttenMix outperforms existing state-of-the-art mixup methods across eight different benchmarks. Additionally, we introduce a new metric based on the human gaze and use this metric to investigate the issue of image-label inconsistency. Project page: \url{https://github.com/morning12138/TdAttenMix}

Figures (6)

• Figure 1: Left: SaliencyMix vs. TdAttenMix. Since SaliencyMix selects to crop the patch with the most salient region, it is distracted by irrelevant dark stone. Our TdAttenMix balances top-down and bottom-up attention and thus picks salient areas consistent with the dog label. Right: Traing time vs. accuracy with Deit-S on ImageNet-1k. TdAttenMix improves performance without the heavy computational overhead.
• Figure 2: The framework of TdAttenMix. (1)Task Adaptive Attention Guided CutMix: compute the task adaptive attention map via manipulating the bottom-up attention using our proposed Top-down Attention Guided Module and then uses the task adaptive attention map to crop the patch. (2)Area-Attention Label Mixing: adjust label mixing based on the ratio of attention and area.
• Figure 3: To simulate the top-down mechanism, we designed the Top-down Attention Guided Module by using the image label as the high-level task information to guide image feature generation, resulting in what we refer to as the "top-down signal." This top-down signal then constrains bottom-up attention to focus on regions related to the image label.
• Figure 4: Top-1 accuracy of mixed data. Prediction is counted as correct if the top-1 prediction belongs to $\{y_A, y_B\}$; Top-2 accuracy is calculated by counting the top-2 predictions are equal to $\{y_A, y_B\}$. $\lambda_r$ indicates the area ratio of $x_A$ in mixed image $x_M$.
• Figure 5: Robustness against occlusion. Model robustness against occlusion with different information loss ratios is studied. 3 patch dropping settings: Random Patch Dropping (left), Salient Patch Dropping (middle), and Non-Salient Patch Dropping (right) are considered.