Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Artificially Generated Visual Scanpath Improves Multi-label Thoracic Disease Classification in Chest X-Ray Images

Ashish Verma, Aupendu Kar, Krishnendu Ghosh, Sobhan Kanti Dhara, Debashis Sen, Prabir Kumar Biswas

TL;DR

The paper addresses the challenge of multi-label thoracic disease classification in Chest X-Rays by learning to mimic radiologists’ gaze through artificial scanpaths. It introduces a two-module framework: a visual scanpath predictor based on an LSTM that generates scanpaths from image features, and a scanpath-guided dynamic classifier that fuses these predicted viewing patterns with visual features via an iterative sequential model with attention. The approach is trained on the REFLACX dataset and evaluated on large-scale MIMIC and cross-dataset CheXpert data, demonstrating that artificial scanpaths yield consistent improvements in AUROC and AUPRC for 14 thoracic findings, both within and across datasets. The work provides a first step toward scanpath-guided automated screening in CXR and highlights potential for improved generalization in medical image analysis, with broad implications for radiology-assisted decision support. Key contributions include (i) a novel iterative predictor network to generate human-like scanpaths, (ii) a scanpath-guided dynamic network with attention for multi-label disease classification, and (iii) substantial cross-dataset performance gains on large CXR collections.

Abstract

Expert radiologists visually scan Chest X-Ray (CXR) images, sequentially fixating on anatomical structures to perform disease diagnosis. An automatic multi-label classifier of diseases in CXR images can benefit by incorporating aspects of the radiologists' approach. Recorded visual scanpaths of radiologists on CXR images can be used for the said purpose. But, such scanpaths are not available for most CXR images, which creates a gap even for modern deep learning based classifiers. This paper proposes to mitigate this gap by generating effective artificial visual scanpaths using a visual scanpath prediction model for CXR images. Further, a multi-class multi-label classifier framework is proposed that uses a generated scanpath and visual image features to classify diseases in CXR images. While the scanpath predictor is based on a recurrent neural network, the multi-label classifier involves a novel iterative sequential model with an attention module. We show that our scanpath predictor generates human-like visual scanpaths. We also demonstrate that the use of artificial visual scanpaths improves multi-class multi-label disease classification results on CXR images. The above observations are made from experiments involving around 0.2 million CXR images from 2 widely-used datasets considering the multi-label classification of 14 pathological findings. Code link: https://github.com/ashishverma03/SDC

Artificially Generated Visual Scanpath Improves Multi-label Thoracic Disease Classification in Chest X-Ray Images

TL;DR

The paper addresses the challenge of multi-label thoracic disease classification in Chest X-Rays by learning to mimic radiologists’ gaze through artificial scanpaths. It introduces a two-module framework: a visual scanpath predictor based on an LSTM that generates scanpaths from image features, and a scanpath-guided dynamic classifier that fuses these predicted viewing patterns with visual features via an iterative sequential model with attention. The approach is trained on the REFLACX dataset and evaluated on large-scale MIMIC and cross-dataset CheXpert data, demonstrating that artificial scanpaths yield consistent improvements in AUROC and AUPRC for 14 thoracic findings, both within and across datasets. The work provides a first step toward scanpath-guided automated screening in CXR and highlights potential for improved generalization in medical image analysis, with broad implications for radiology-assisted decision support. Key contributions include (i) a novel iterative predictor network to generate human-like scanpaths, (ii) a scanpath-guided dynamic network with attention for multi-label disease classification, and (iii) substantial cross-dataset performance gains on large CXR collections.

Abstract

Expert radiologists visually scan Chest X-Ray (CXR) images, sequentially fixating on anatomical structures to perform disease diagnosis. An automatic multi-label classifier of diseases in CXR images can benefit by incorporating aspects of the radiologists' approach. Recorded visual scanpaths of radiologists on CXR images can be used for the said purpose. But, such scanpaths are not available for most CXR images, which creates a gap even for modern deep learning based classifiers. This paper proposes to mitigate this gap by generating effective artificial visual scanpaths using a visual scanpath prediction model for CXR images. Further, a multi-class multi-label classifier framework is proposed that uses a generated scanpath and visual image features to classify diseases in CXR images. While the scanpath predictor is based on a recurrent neural network, the multi-label classifier involves a novel iterative sequential model with an attention module. We show that our scanpath predictor generates human-like visual scanpaths. We also demonstrate that the use of artificial visual scanpaths improves multi-class multi-label disease classification results on CXR images. The above observations are made from experiments involving around 0.2 million CXR images from 2 widely-used datasets considering the multi-label classification of 14 pathological findings. Code link: https://github.com/ashishverma03/SDC

Paper Structure

This paper contains 15 sections, 7 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Automatic Disease Classification in Chest X-ray
  4. Scanpath and Visual Attention
  5. The Proposed Deep Models
  6. Scanpath Prediction Model
  7. Visual scanpath-based dynamic network for multi-class multi-label disease classification
  8. Results and Discussion
  9. Datasets
  10. Experimental Settings
  11. Performance of Scanpath Prediction
  12. Performance of Scanpath based Classification
  13. Ablation Study
  14. Conclusion
  15. Aknowledgement

Figures (3)

  • Figure 1: The proposed visual scanpath predictor network is trained on the recorded visual scanpaths from radiologists on CXR images. The spatial partitioning of images and forming the dictionary $\mathcal{D}$ convert the task of pixel-level prediction of fixation location to iterative patch classification. The iterative predictions of patches as fixation locations generate a visual scanpath on a given CXR image.
  • Figure 2: Visual scanpath-based dynamic network for multi-class multi-label disease classification. The network comprises of a CNN as a feature extractor and an iterative sequential model (ISM). In one branch of the network semantic feature map is passed through the global average pooling layer to produce visual features. The other branch employs ISM with an attention module on the sequence of visual features corresponding to a visual scanpath to give a viewing-pattern feature. The visual feature and viewing pattern feature are concatenated and fed to the classifier.
  • Figure 3: Performance of proposed visual scanpath prediction model compared to two radiologists (Radiologist-1 and Radiologist-2), GenLSTM, Random and CLE on two images (Image-1 and Image-2). The first and third rows plot the scanpaths with corresponding ScanMatch values on Image-1 and Image-2. The second and fourth rows plot the heatmaps corresponding to the scanpaths plotted in the first and third rows, respectively.