A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answering

TL;DR

This paper tackles MVQA by addressing limited medical information extraction from questions and shallow visual-text reasoning. It introduces WSDAN, which combines a Transformer with sentence embedding (TSE) to produce double-embedding question representations and a cascade of Dual-Attention Learning (DAL) modules to learn self- and guided-attention for tight cross-modal fusion, using ResNet-152 visual features and a classifier for answers. Empirical results on VQA-MED 2019 and VQA-RAD show state-of-the-art performance, with ablations confirming the effectiveness of TSE and DAL in improving textual understanding and visual reasoning; Grad-CAM analyses corroborate focused cross-modal attention. The work advances MVQA by enabling richer medical-question representations and finer-grained co-attention, with potential to support computer-aided diagnosis and clinical decision support.

Abstract

Research in medical visual question answering (MVQA) can contribute to the development of computeraided diagnosis. MVQA is a task that aims to predict accurate and convincing answers based on given medical images and associated natural language questions. This task requires extracting medical knowledge-rich feature content and making fine-grained understandings of them. Therefore, constructing an effective feature extraction and understanding scheme are keys to modeling. Existing MVQA question extraction schemes mainly focus on word information, ignoring medical information in the text. Meanwhile, some visual and textual feature understanding schemes cannot effectively capture the correlation between regions and keywords for reasonable visual reasoning. In this study, a dual-attention learning network with word and sentence embedding (WSDAN) is proposed. We design a module, transformer with sentence embedding (TSE), to extract a double embedding representation of questions containing keywords and medical information. A dualattention learning (DAL) module consisting of self-attention and guided attention is proposed to model intensive intramodal and intermodal interactions. With multiple DAL modules (DALs), learning visual and textual co-attention can increase the granularity of understanding and improve visual reasoning. Experimental results on the ImageCLEF 2019 VQA-MED (VQA-MED 2019) and VQA-RAD datasets demonstrate that our proposed method outperforms previous state-of-the-art methods. According to the ablation studies and Grad-CAM maps, WSDAN can extract rich textual information and has strong visual reasoning ability.

Figures (7)

• Figure 1: The proposed WSDAN framework. The ResNet-152 is used to extract image features, $V$. TSE is used to extract the question features, $Q$. The $L$ DALs form the fusion module, and the fusion vector is obtained with the help of a weighted combination. The classifier consists of MLP and softmax.
• Figure 2: Question Encoder: TSE fuses word and sentence embedding to obtain a double embedding representation of the question.
• Figure 3: Details of our proposed Dual-Attention Learning (DAL) module. Multi-head attention in the first and second layers is used to learn self-attention and guided attention, respectively.
• Figure 4: The confusion matrix of the Modality category.
• Figure 5: The confusion matrix of the Plane category.