Gaze-directed Vision GNN for Mitigating Shortcut Learning in Medical Image

TL;DR

This work addresses shortcut learning in medical image analysis by leveraging radiologists' gaze as a prior. It introduces GD-ViG, an end-to-end framework with a gaze map generator (GMG) and a gaze-directed classifier (GDC) that uses gaze-distance based graph construction to focus on disease-relevant foregrounds, eliminating the need for real gaze data at inference. GMG learns gaze patterns via a loss $ L_{GMG} = \frac{1}{HW} \sum (gm_i - g_i)^2 $, while GDC builds a graph with distance dist(x_i, x_j) = ||x_i - x_j||^2 + lambda_g ||gm_i - gm_j||^2 * gm_i and optimizes $ L = L_{GMG} + lambda_c L_{GDC} $. Experiments on SIIM-ACR and EGD-CXR show state-of-the-art accuracy and improvements in AUC and F1, with ablations confirming the benefits of combining CNN and GNN-based gaze map generation and of gaze-distance fusion for robust, interpretable graph structures. The approach enhances practical applicability by removing the need for real gaze during inference and improves reliability in detecting subtle and sparse disease indicators in chest X-rays.

Abstract

Deep neural networks have demonstrated remarkable performance in medical image analysis. However, its susceptibility to spurious correlations due to shortcut learning raises concerns about network interpretability and reliability. Furthermore, shortcut learning is exacerbated in medical contexts where disease indicators are often subtle and sparse. In this paper, we propose a novel gaze-directed Vision GNN (called GD-ViG) to leverage the visual patterns of radiologists from gaze as expert knowledge, directing the network toward disease-relevant regions, and thereby mitigating shortcut learning. GD-ViG consists of a gaze map generator (GMG) and a gaze-directed classifier (GDC). Combining the global modelling ability of GNNs with the locality of CNNs, GMG generates the gaze map based on radiologists' visual patterns. Notably, it eliminates the need for real gaze data during inference, enhancing the network's practical applicability. Utilizing gaze as the expert knowledge, the GDC directs the construction of graph structures by incorporating both feature distances and gaze distances, enabling the network to focus on disease-relevant foregrounds. Thereby avoiding shortcut learning and improving the network's interpretability. The experiments on two public medical image datasets demonstrate that GD-ViG outperforms the state-of-the-art methods, and effectively mitigates shortcut learning. Our code is available at https://github.com/SX-SS/GD-ViG.

Figures (3)

• Figure 1: Overview of the proposed framework consisting of two subnets: 1) Gaze map generator; 2) Gaze-directed classifier.
• Figure 2: Comparison results of attention map of different methods on SIIM-ACR dataset. In the first column, the pneumothorax region is marked in red mask. Networks prone to shortcut learning are indicated by red arrows, while blue arrows denote regions where networks accurately concentrate on disease-specific areas.
• Figure 3: Visualization of the graph structure based on different distances. The red regions and the blue regions represent the nodes that are eliminated and the disease-relevant nodes that are retained after distance fusion, respectively.