Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Bilinear Attention-based Fusion for Medical Visual Question Answering

Zhilin Zhang, Jie Wang, Zhanghao Qin, Ruiqi Zhu, Xiaoliang Gong

TL;DR

The paper tackles MedVQA by addressing the high computational cost of state-of-the-art fusion models. It introduces OMniBAN, a fusion framework that combines Orthogonality loss, Multi-head self-attention, and Bilinear Attention to efficiently fuse frozen visual and textual encoders (BiomedCLIP and BioBERT). Empirical results on VQA-RAD and SLAKE-EN show that OMniBAN achieves comparable overall accuracy to Transformer-based fusion while substantially reducing parameters (approximately 2/3) and FLOPs (approximately 1/4), with notable gains on closed-ended questions. These findings suggest that bilinear attention with orthogonality-regularized glimpses can provide a practical, efficient alternative for MedVQA in resource-constrained clinical environments, and point to further exploration with more encoders and medical knowledge integration.

Abstract

Medical Visual Question Answering (MedVQA) has attracted growing interest at the intersection of medical image understanding and natural language processing for clinical applications. By interpreting medical images and providing precise answers to relevant clinical inquiries, MedVQA has the potential to support diagnostic decision-making and reduce workload across various fields like radiology. While recent approaches rely heavily on unified large pre-trained Visual-Language Models, research on more efficient fusion mechanisms remains relatively limited in this domain. In this paper, we introduce a fusion model, OMniBAN, that integrates Orthogonality loss, Multi-head attention, and a Bilinear Attention Network to achieve high computational efficiency as well as solid performance. We conduct comprehensive experiments and demonstrate how bilinear attention fusion can approximate the performance of larger fusion models like cross-modal Transformer. Our results show that OMniBAN requires fewer parameters (approximately 2/3 of Transformer-based Co-Attention) and substantially lower FLOPs (approximately 1/4), while achieving comparable overall performance and even slight improvements on closed-ended questions on two key MedVQA benchmarks. This balance between efficiency and accuracy suggests that OMniBAN could be a viable option for real-world medical image question answering, where computational resources are often constrained.

Efficient Bilinear Attention-based Fusion for Medical Visual Question Answering

TL;DR

The paper tackles MedVQA by addressing the high computational cost of state-of-the-art fusion models. It introduces OMniBAN, a fusion framework that combines Orthogonality loss, Multi-head self-attention, and Bilinear Attention to efficiently fuse frozen visual and textual encoders (BiomedCLIP and BioBERT). Empirical results on VQA-RAD and SLAKE-EN show that OMniBAN achieves comparable overall accuracy to Transformer-based fusion while substantially reducing parameters (approximately 2/3) and FLOPs (approximately 1/4), with notable gains on closed-ended questions. These findings suggest that bilinear attention with orthogonality-regularized glimpses can provide a practical, efficient alternative for MedVQA in resource-constrained clinical environments, and point to further exploration with more encoders and medical knowledge integration.

Abstract

Medical Visual Question Answering (MedVQA) has attracted growing interest at the intersection of medical image understanding and natural language processing for clinical applications. By interpreting medical images and providing precise answers to relevant clinical inquiries, MedVQA has the potential to support diagnostic decision-making and reduce workload across various fields like radiology. While recent approaches rely heavily on unified large pre-trained Visual-Language Models, research on more efficient fusion mechanisms remains relatively limited in this domain. In this paper, we introduce a fusion model, OMniBAN, that integrates Orthogonality loss, Multi-head attention, and a Bilinear Attention Network to achieve high computational efficiency as well as solid performance. We conduct comprehensive experiments and demonstrate how bilinear attention fusion can approximate the performance of larger fusion models like cross-modal Transformer. Our results show that OMniBAN requires fewer parameters (approximately 2/3 of Transformer-based Co-Attention) and substantially lower FLOPs (approximately 1/4), while achieving comparable overall performance and even slight improvements on closed-ended questions on two key MedVQA benchmarks. This balance between efficiency and accuracy suggests that OMniBAN could be a viable option for real-world medical image question answering, where computational resources are often constrained.

Paper Structure

This paper contains 19 sections, 10 equations, 1 figure, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Medical Visual Question Answering
  4. Multi-modal Fusion
  5. Method
  6. Problem Formulation
  7. Multi-modal Feature Extraction
  8. Orthogonal Multi-head Bilinear Attention Network
  9. Intra-modal Feature Attention
  10. Cross-modal Bilinear Attention
  11. Orthogonality Loss
  12. Classifier and Loss Function
  13. Theoretical Complexity Analysis
  14. Experiments
  15. Datasets and Metrics
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Overview of our proposed Orthogonal Multi-head Bilinear Attention Network (OMniBAN). The frozen visual and textual backbones encode the input image and question independently, and then the core OMniBAN fusion module fuses these features using multi-head attention for within-modality refinement and bilinear attention to capture cross-modal interactions. Orthogonality loss is adopted to encourage diverse attention patterns among glimpses during training.