DGSAN: Dual-Graph Spatiotemporal Attention Network for Pulmonary Nodule Malignancy Prediction

TL;DR

The paper tackles dynamic pulmonary nodule malignancy prediction by integrating longitudinal CT scans with clinical data. It introduces a Global-Local Feature Encoder to capture multi-scale texture and context, a dual-graph framework to model intra- and inter-modal relationships, and a Hierarchical Cross-Modal Graph Fusion Module for progressive multimodal alignment. Key contributions include the NLST-cmst dataset release, a lightweight yet powerful graph-based fusion architecture, and state-of-the-art performance on NLST-cmst and external CLST data with notable efficiency. These advances enable more accurate and robust dynamic nodule assessment, with potential to enhance clinical decision-making and research in multimodal medical imaging.

Abstract

Lung cancer continues to be the leading cause of cancer-related deaths globally. Early detection and diagnosis of pulmonary nodules are essential for improving patient survival rates. Although previous research has integrated multimodal and multi-temporal information, outperforming single modality and single time point, the fusion methods are limited to inefficient vector concatenation and simple mutual attention, highlighting the need for more effective multimodal information fusion. To address these challenges, we introduce a Dual-Graph Spatiotemporal Attention Network, which leverages temporal variations and multimodal data to enhance the accuracy of predictions. Our methodology involves developing a Global-Local Feature Encoder to better capture the local, global, and fused characteristics of pulmonary nodules. Additionally, a Dual-Graph Construction method organizes multimodal features into inter-modal and intra-modal graphs. Furthermore, a Hierarchical Cross-Modal Graph Fusion Module is introduced to refine feature integration. We also compiled a novel multimodal dataset named the NLST-cmst dataset as a comprehensive source of support for related research. Our extensive experiments, conducted on both the NLST-cmst and curated CSTL-derived datasets, demonstrate that our DGSAN significantly outperforms state-of-the-art methods in classifying pulmonary nodules with exceptional computational efficiency.

Figures (4)

• Figure 1: Overall framework of the proposed DGSAN, which comprises three parts: (A) Multi-modal Feature Extraction, designed to fuse local and global information and generate high-dimensional spatiotemporal representations, (B) Hierarchical Graph Construction, builds intra- and inter-modal graphs based on imaging and clinical features, using graph attention to model complex spatial–temporal and cross-modal dependencies, and (C) Graph Feature Fusion, utilizes a “self → cross → self” attention mechanism to align multimodal information and produce unified representations for malignancy prediction.
• Figure 2: Illustration of correlations in pulmonary nodule diagnosis. (a) Inter-modal correlations, (b) Intra-modal correlations, (c) A relationship diagram with inter-modal and intra-modal information.
• Figure 3: Five modal graph construction schemes. From left to right: Separate graphs for each modality, No local/global features used, No feature fusion, Separate graphs for relevant features at two-time points, and Our custom scheme.
• Figure 4: ROC curves comparison between DGSAN and other methods on the NLST-cmst dataset.