Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptions

Jun Li, Che Liu, Wenjia Bai, Rossella Arcucci, Cosmin I. Bercea, Julia A. Schnabel

TL;DR

The paper tackles abnormality grounding in medical VLMs, where abstract medical terms hinder direct visual-to-text alignment. It introduces decomposed knowledge descriptions that map diseases to core visual attributes (shape, location, density, color) and trains a compact Florence-2 base (0.23B) on a small, targeted dataset to perform localization via autoregressive token generation. The approach, validated on VinDr-CXR and PadChest-GR, achieves competitive or superior performance compared to much larger models, with strong zero-shot generalization and substantial ablation gains from knowledge-enhanced prompts. This work demonstrates that knowledge-enhanced prompts can dramatically improve medical visual grounding while reducing data and compute requirements, enabling better applicability in low-data clinical contexts.

Abstract

Visual Language Models (VLMs) have demonstrated impressive capabilities in visual grounding tasks. However, their effectiveness in the medical domain, particularly for abnormality detection and localization within medical images, remains underexplored. A major challenge is the complex and abstract nature of medical terminology, which makes it difficult to directly associate pathological anomaly terms with their corresponding visual features. In this work, we introduce a novel approach to enhance VLM performance in medical abnormality detection and localization by leveraging decomposed medical knowledge. Instead of directly prompting models to recognize specific abnormalities, we focus on breaking down medical concepts into fundamental attributes and common visual patterns. This strategy promotes a stronger alignment between textual descriptions and visual features, improving both the recognition and localization of abnormalities in medical images.We evaluate our method on the 0.23B Florence-2 base model and demonstrate that it achieves comparable performance in abnormality grounding to significantly larger 7B LLaVA-based medical VLMs, despite being trained on only 1.5% of the data used for such models. Experimental results also demonstrate the effectiveness of our approach in both known and previously unseen abnormalities, suggesting its strong generalization capabilities.

Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptions

TL;DR

The paper tackles abnormality grounding in medical VLMs, where abstract medical terms hinder direct visual-to-text alignment. It introduces decomposed knowledge descriptions that map diseases to core visual attributes (shape, location, density, color) and trains a compact Florence-2 base (0.23B) on a small, targeted dataset to perform localization via autoregressive token generation. The approach, validated on VinDr-CXR and PadChest-GR, achieves competitive or superior performance compared to much larger models, with strong zero-shot generalization and substantial ablation gains from knowledge-enhanced prompts. This work demonstrates that knowledge-enhanced prompts can dramatically improve medical visual grounding while reducing data and compute requirements, enabling better applicability in low-data clinical contexts.

Abstract

Visual Language Models (VLMs) have demonstrated impressive capabilities in visual grounding tasks. However, their effectiveness in the medical domain, particularly for abnormality detection and localization within medical images, remains underexplored. A major challenge is the complex and abstract nature of medical terminology, which makes it difficult to directly associate pathological anomaly terms with their corresponding visual features. In this work, we introduce a novel approach to enhance VLM performance in medical abnormality detection and localization by leveraging decomposed medical knowledge. Instead of directly prompting models to recognize specific abnormalities, we focus on breaking down medical concepts into fundamental attributes and common visual patterns. This strategy promotes a stronger alignment between textual descriptions and visual features, improving both the recognition and localization of abnormalities in medical images.We evaluate our method on the 0.23B Florence-2 base model and demonstrate that it achieves comparable performance in abnormality grounding to significantly larger 7B LLaVA-based medical VLMs, despite being trained on only 1.5% of the data used for such models. Experimental results also demonstrate the effectiveness of our approach in both known and previously unseen abnormalities, suggesting its strong generalization capabilities.

Paper Structure

This paper contains 10 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Problem Setup.
  4. Decomposed Knowledge Description
  5. Architecture.
  6. Experiments
  7. Results
  8. Comparison with SOTA
  9. Ablation Study
  10. Discussion and Conclusion

Figures (3)

  • Figure 1: Overview of our approach. We train a 0.23B model on just 16,087 samples (1.5% of the data) and achieve similar or better results than the 7B RadVLM, pre-trained on 1 million samples, by using text descriptions that highlight key visual features of abnormalities.
  • Figure 2: Overview of our method. (A) shows the pipeline for obtaining decomposed knowledge descriptions, (B) presents the model architecture and training process for the abnormality grounding task.
  • Figure 3: Performance for each disease class, with the y-axis representing the RoDeo total metric. Our method achieves first place in 14 out of 21 diseases from the VinDr-CXR dataset and 3 out of 6 known diseases from the PadChest-GR dataset. The best performances are highlighted in the callout.