Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Describe Anything in Medical Images

Xi Xiao, Yunbei Zhang, Thanh-Huy Nguyen, Ba-Thinh Lam, Janet Wang, Lin Zhao, Jihun Hamm, Tianyang Wang, Xingjian Li, Xiao Wang, Hao Xu, Tianming Liu, Min Xu

TL;DR

The paper tackles the lack of region-level captioning for medical images by extending the Describe Anything Model (DAM) to the medical domain with MedDAM, a framework driven by expert-designed prompts, a flexible ROI pipeline, and a reference-free evaluation benchmark. It introduces MedDLC-score, an attribute-level verification metric, alongside an LLM-based judge (LLM-score) to assess both factuality and fluency in a zero-shot setting across VinDr-CXR, LIDC-IDRI, and SkinCon. Empirical results show MedDAM achieving superior MedDLC-score and strong region-specific outputs compared to several strong adaptable large vision-language models, highlighting the importance of region-level semantic alignment in medical image understanding. The work advances clinical vision-language integration by enabling accurate, region-focused descriptions without requiring ground-truth region captions and outlines future directions for broader modalities and knowledge-grounded prompting.

Abstract

Localized image captioning has made significant progress with models like the Describe Anything Model (DAM), which can generate detailed region-specific descriptions without explicit region-text supervision. However, such capabilities have yet to be widely applied to specialized domains like medical imaging, where diagnostic interpretation relies on subtle regional findings rather than global understanding. To mitigate this gap, we propose MedDAM, the first comprehensive framework leveraging large vision-language models for region-specific captioning in medical images. MedDAM employs medical expert-designed prompts tailored to specific imaging modalities and establishes a robust evaluation benchmark comprising a customized assessment protocol, data pre-processing pipeline, and specialized QA template library. This benchmark evaluates both MedDAM and other adaptable large vision-language models, focusing on clinical factuality through attribute-level verification tasks, thereby circumventing the absence of ground-truth region-caption pairs in medical datasets. Extensive experiments on the VinDr-CXR, LIDC-IDRI, and SkinCon datasets demonstrate MedDAM's superiority over leading peers (including GPT-4o, Claude 3.7 Sonnet, LLaMA-3.2 Vision, Qwen2.5-VL, GPT-4Rol, and OMG-LLaVA) in the task, revealing the importance of region-level semantic alignment in medical image understanding and establishing MedDAM as a promising foundation for clinical vision-language integration.

Describe Anything in Medical Images

TL;DR

The paper tackles the lack of region-level captioning for medical images by extending the Describe Anything Model (DAM) to the medical domain with MedDAM, a framework driven by expert-designed prompts, a flexible ROI pipeline, and a reference-free evaluation benchmark. It introduces MedDLC-score, an attribute-level verification metric, alongside an LLM-based judge (LLM-score) to assess both factuality and fluency in a zero-shot setting across VinDr-CXR, LIDC-IDRI, and SkinCon. Empirical results show MedDAM achieving superior MedDLC-score and strong region-specific outputs compared to several strong adaptable large vision-language models, highlighting the importance of region-level semantic alignment in medical image understanding. The work advances clinical vision-language integration by enabling accurate, region-focused descriptions without requiring ground-truth region captions and outlines future directions for broader modalities and knowledge-grounded prompting.

Abstract

Localized image captioning has made significant progress with models like the Describe Anything Model (DAM), which can generate detailed region-specific descriptions without explicit region-text supervision. However, such capabilities have yet to be widely applied to specialized domains like medical imaging, where diagnostic interpretation relies on subtle regional findings rather than global understanding. To mitigate this gap, we propose MedDAM, the first comprehensive framework leveraging large vision-language models for region-specific captioning in medical images. MedDAM employs medical expert-designed prompts tailored to specific imaging modalities and establishes a robust evaluation benchmark comprising a customized assessment protocol, data pre-processing pipeline, and specialized QA template library. This benchmark evaluates both MedDAM and other adaptable large vision-language models, focusing on clinical factuality through attribute-level verification tasks, thereby circumventing the absence of ground-truth region-caption pairs in medical datasets. Extensive experiments on the VinDr-CXR, LIDC-IDRI, and SkinCon datasets demonstrate MedDAM's superiority over leading peers (including GPT-4o, Claude 3.7 Sonnet, LLaMA-3.2 Vision, Qwen2.5-VL, GPT-4Rol, and OMG-LLaVA) in the task, revealing the importance of region-level semantic alignment in medical image understanding and establishing MedDAM as a promising foundation for clinical vision-language integration.
Paper Structure (12 sections, 3 figures, 4 tables)

This paper contains 12 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Localized Image Captioning
  4. Medical Image Captioning
  5. Framework and Evaluation
  6. Datasets
  7. Region Sampling and Prompt Construction
  8. Attribute Question Construction
  9. Baselines and Evaluation Metrics
  10. Region-specific Prompting
  11. Main Results
  12. Conclusion and Future Work

Figures (3)

  • Figure 1: Architecture of MedDAM. MedDAM extends the recent breakthrough, i.e., Describe Anything framework lian2025anythingdetailedlocalizedimage, to medical image understanding. A clinically focused region and its binary mask are used to generate a focal crop, which is embedded along with the full image, fusing global and regional features via gated cross-attention, while structured prompt tokens encode clinical objectives. Then, the resulting features are fed into a LLM to generate region-specific captions.
  • Figure 2: An example evaluation pipeline (i.e., calculating MedDLC-score). (a) A region of interest is marked in a chest X-ray image and then used as input to MedDAM, prompted by a task-specific instruction. (b) The model generates a region-specific caption describing the abnormality within the marked region. (c) A question-answering task is set up to verify the factual accuracy and localization consistency of the generated caption, based on domain-specific attributes. (d) An LLM-based evaluator assigns a correctness label to the answer, and the model receives a score if the description matches the ground-truth semantic attributes. This framework enables reference-free benchmarking of fine-grained regional captioning performance on medical image understanding. Although both the widely used LLM-score and our MedDLC-score involve a LLM Judge, the former is more effective for natural vision-language scenarios while the latter is tailored to the proposed task in medical domain.
  • Figure 3: MedDAM-prompt Template. This prompt guides the model to produce region-specific, clinically accurate descriptions by incorporating task constraints such as anatomical focus, output format, and information grounding. It is essential for adapting general captioning models like DAM to medical images.