WoLF: Wide-scope Large Language Model Framework for CXR Understanding

TL;DR

WoLF addresses key gaps in CXR understanding by integrating health records through Health-specific Instruction Tuning (HIT), reorganizing CXR reports by anatomy using Anatomy-Specific Knowledge decoupling (ASK) with Anatomy-localizing Masked Attention (AMA), and evaluating generative LLM performance with a multi-faceted AI-evaluation protocol. The framework unifies data reformulation, two-stage training, and AI-based evaluation to enable accurate VQA and anatomy-aligned report generation on CXR data such as MIMIC-CXR and IU-Xray. Empirical results demonstrate state-of-the-art performance on VQA and report generation, with significant gains in AI-evaluation metrics (up to +9.47 percentage points) and BLEU-1 (up to +7.3 points). The approach promises enhanced clinical reliability and lays groundwork for integrating real-time EHR retrieval to further bolster decision support in radiology.

Abstract

Significant methodological strides have been made toward Chest X-ray (CXR) understanding via modern vision-language models (VLMs), demonstrating impressive Visual Question Answering (VQA) and CXR report generation abilities. However, existing CXR understanding frameworks still possess several procedural caveats. (1) Previous methods solely use CXR reports, which are insufficient for comprehensive Visual Question Answering (VQA), especially when additional health-related data like medication history and prior diagnoses are needed. (2) Previous methods use raw CXR reports, which are often arbitrarily structured. While modern language models can understand various text formats, restructuring reports for clearer, organized anatomy-based information could enhance their usefulness. (3) Current evaluation methods for CXR-VQA primarily emphasize linguistic correctness, lacking the capability to offer nuanced assessments of the generated answers. In this work, to address the aforementioned caveats, we introduce WoLF, a Wide-scope Large Language Model Framework for CXR understanding. To resolve (1), we capture multi-faceted records of patients, which are utilized for accurate diagnoses in real-world clinical scenarios. Specifically, we adopt the Electronic Health Records (EHR) to generate instruction-following data suited for CXR understanding. Regarding (2), we enhance report generation performance by decoupling knowledge in CXR reports based on anatomical structure even within the attention step via masked attention. To address (3), we introduce an AI-evaluation protocol optimized for assessing the capabilities of LLM. Through extensive experimental validations, WoLF demonstrates superior performance over other models on MIMIC-CXR in the AI-evaluation arena about VQA (up to +9.47%p mean score) and by metrics about report generation (+7.3%p BLEU-1).

Figures (8)

• Figure 1: Comparisons with other models for VQA scenario given a CXR image. Green thumbs indicate the quality of the response is good (accurate, helpful), while red thumbs indicate bad (inaccurate, evasive), with respect to target answers.
• Figure 1: Evaluation prompts for scoring (left) and prioritizing (right). Preamble and evaluation criteria are common in both scoring and prioritizing prompts. The two prompts, each for AI-evaluation (left) and for its ablation on Win-rate (right), differ only in evaluation steps.
• Figure 2: Data generation overview of HIT and ASK: (a) We generate health-specific instruction-following dataset. In (a), Cyan and orange sequences are queries about EHR and findings in CXR respectively. (b) We reorganize original CXR reports into sequences of anatomy-specific structures through the use of a knowledge graph, $G$.
• Figure 2: Qualitative results of visual question-answering scenarios. As shown in (a) and (b), the model can be fed with patient histories and medication details from EHRs. WoLF utilizes these external contexts to deliver more accurate responses. (c) shows question-answering when there are no findings. The model answered correctly that no disease could be found, without causing any hallucinations.
• Figure 3: Overview of the training phase. (a) The input embedding ${{\bf H}}_{(\cdot)}$ consists of visual embedding from the adapter and language embeddings from $t$-turn conversations generated by HIT. Cyan and orange sequences are queries about EHR and findings in CXR respectively. (b) For an anatomical structure $o_m$, its sequence embedding is denoted by ${{\bf H}}^{o_m}$. Organized CXR reports by ASK are utilized as input for training.