Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework

TL;DR

This work addresses misalignment in medical vision-language pre-training by decomposing disease descriptions into fine-grained visual aspects using a knowledge base, LLMs, and clinician input. The authors introduce MAVL, an aspect-based multi-aspect vision-language pre-training framework, and an aspect-oriented dual-head Transformer that jointly learns a contrastive representation for unseen diseases and a supervised representation for seen diseases. The approach leverages a semi-automatic pipeline to generate eight visual aspects per disease (plus a detailed description) and performs aspect-wise image-text matching, improving zero-shot and few-shot transfer, as well as visual grounding. Across seven chest X-ray datasets, MAVL delivers substantial gains in both zero-shot and fine-tuning settings and demonstrates enhanced localization of abnormalities, indicating strong generalization to novel and rare diseases and potential for improved clinical deployment.

Abstract

Medical vision language pre-training (VLP) has emerged as a frontier of research, enabling zero-shot pathological recognition by comparing the query image with the textual descriptions for each disease. Due to the complex semantics of biomedical texts, current methods struggle to align medical images with key pathological findings in unstructured reports. This leads to the misalignment with the target disease's textual representation. In this paper, we introduce a novel VLP framework designed to dissect disease descriptions into their fundamental aspects, leveraging prior knowledge about the visual manifestations of pathologies. This is achieved by consulting a large language model and medical experts. Integrating a Transformer module, our approach aligns an input image with the diverse elements of a disease, generating aspect-centric image representations. By consolidating the matches from each aspect, we improve the compatibility between an image and its associated disease. Additionally, capitalizing on the aspect-oriented representations, we present a dual-head Transformer tailored to process known and unknown diseases, optimizing the comprehensive detection efficacy. Conducting experiments on seven downstream datasets, ours improves the accuracy of recent methods by up to 8.56% and 17.26% for seen and unseen categories, respectively. Our code is released at https://github.com/HieuPhan33/MAVL.

Figures (6)

• Figure 1: Predictions of CheXzero tiu2022expert (Left), a strong CLIP-like model, and our multi-aspect matching model (Right).Edema and Covid-19 belong to the domain of lung diseases, while cardiomegaly is a heart disease. CheXzero tiu2022expert misaligns the image feature with the target Covid-19, while it over-aligns with edema. We leverage the medical knowledge base to decompose disease terms into distinct visual components, enhancing the image alignment with the representations of the target disease.
• Figure 2: Illustrations of the three VLP paradigms: image-report matching bannur2023learningboecking2022makingtiu2022expert (red arrow), image-disease definition matching wu2023medklip (orange arrow), and our proposed fine-grained image-aspect matching (green arrow).
• Figure 3: Pipeline to extract visual aspect's descriptions of diseases mentioned in the pre-training MIMIC dataset johnson2019mimic.
• Figure 4: Multi-aspect vision-language pre-training framework (MAVL) decomposes diseases into a set of shared visual aspects using LLM and prior knowledge from two medical experts. An aspect-oriented dual-head Transformer (A) visually searches for the queried visual aspects in the image and maximizes detection accuracy of both unseen and seen diseases via two learning signals. The contrastive head (B) captures generalizable features and performs zero-shot classification of unseen diseases by comparing visual aspects between the target disease and the healthy category. The supervised head (C) captures discriminative features to classify fine-grained seen diseases.
• Figure 5: Visual grounding prediction scores of RSNA Pneumonia (Top) and Covid-19 (Bottom).