Enhancing Human-Computer Interaction in Chest X-ray Analysis using Vision and Language Model with Eye Gaze Patterns

TL;DR

The paper addresses the limited human-computer interaction in chest X-ray analysis by integrating radiologists' eye gaze heatmaps into Vision-Language Models, overlaying gaze-derived heatmaps on CXRs and fine-tuning with gaze data. It evaluates four clinical tasks—Report Automation (GEN/SUM), Error Detection (ERR), Differential Diagnosis (DDx), and Visual Question Answering (VQA)—using multiple model variants, datasets, and zero-shot prompts. Key findings show that eye gaze improves performance across tasks, most notably in DDx, and that domain-specific fine-tuning with gaze data yields substantial gains, while larger models do not always translate to better results. The work demonstrates a promising path toward human-centered AI in CAD with potential extensions to other imaging modalities and CAD tasks, highlighting the value of incorporating radiologists' attention into multimodal models.

Abstract

Recent advancements in Computer Assisted Diagnosis have shown promising performance in medical imaging tasks, particularly in chest X-ray analysis. However, the interaction between these models and radiologists has been primarily limited to input images. This work proposes a novel approach to enhance human-computer interaction in chest X-ray analysis using Vision-Language Models (VLMs) enhanced with radiologists' attention by incorporating eye gaze data alongside textual prompts. Our approach leverages heatmaps generated from eye gaze data, overlaying them onto medical images to highlight areas of intense radiologist's focus during chest X-ray evaluation. We evaluate this methodology in tasks such as visual question answering, chest X-ray report automation, error detection, and differential diagnosis. Our results demonstrate the inclusion of eye gaze information significantly enhances the accuracy of chest X-ray analysis. Also, the impact of eye gaze on fine-tuning was confirmed as it outperformed other medical VLMs in all tasks except visual question answering. This work marks the potential of leveraging both the VLM's capabilities and the radiologist's domain knowledge to improve the capabilities of AI models in medical imaging, paving a novel way for Computer Assisted Diagnosis with a human-centred AI.

Figures (1)

• Figure 1: Overview of Enhancing Human-Computer Interaction in Chest X-ray Analysis using Vision and Language Model with Eye Gaze Patterns.