Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring and Aligning Abstraction in Vision-Language Models with Medical Taxonomies

Ben Schaper, Maxime Di Folco, Bernhard Kainz, Julia A. Schnabel, Cosmin I. Bercea

TL;DR

This work tackles the gap between flat performance and clinical safety in vision–language models for chest X-ray classification by introducing Catastrophic Abstraction Errors ($CAE$) and hierarchical evaluation with a 117-node PadChest-GR taxonomy. It benchmarks SOTA VLMs, defines CAE, and proposes risk-constrained thresholding and taxonomy-aware fine-tuning with Radial Embeddings to reduce cross-branch errors. The results show substantial cross-branch errors in zero-shot settings, which are mitigated to below ~2.4% CAE with thresholding and further minimized while preserving competitive $F_1$ through taxonomy-aligned fine-tuning; Kendall’s rank correlation confirms improved representation alignment with the taxonomy ($\tau=0.86$). The findings highlight the importance of hierarchical evaluation and representation-level alignment for safer, clinically meaningful deployment of multimodal models in radiology.

Abstract

Vision-Language Models show strong zero-shot performance for chest X-ray classification, but standard flat metrics fail to distinguish between clinically minor and severe errors. This work investigates how to quantify and mitigate abstraction errors by leveraging medical taxonomies. We benchmark several state-of-the-art VLMs using hierarchical metrics and introduce Catastrophic Abstraction Errors to capture cross-branch mistakes. Our results reveal substantial misalignment of VLMs with clinical taxonomies despite high flat performance. To address this, we propose risk-constrained thresholding and taxonomy-aware fine-tuning with radial embeddings, which reduce severe abstraction errors to below 2 per cent while maintaining competitive performance. These findings highlight the importance of hierarchical evaluation and representation-level alignment for safer and more clinically meaningful deployment of VLMs.

Measuring and Aligning Abstraction in Vision-Language Models with Medical Taxonomies

TL;DR

This work tackles the gap between flat performance and clinical safety in vision–language models for chest X-ray classification by introducing Catastrophic Abstraction Errors () and hierarchical evaluation with a 117-node PadChest-GR taxonomy. It benchmarks SOTA VLMs, defines CAE, and proposes risk-constrained thresholding and taxonomy-aware fine-tuning with Radial Embeddings to reduce cross-branch errors. The results show substantial cross-branch errors in zero-shot settings, which are mitigated to below ~2.4% CAE with thresholding and further minimized while preserving competitive through taxonomy-aligned fine-tuning; Kendall’s rank correlation confirms improved representation alignment with the taxonomy (). The findings highlight the importance of hierarchical evaluation and representation-level alignment for safer, clinically meaningful deployment of multimodal models in radiology.

Abstract

Vision-Language Models show strong zero-shot performance for chest X-ray classification, but standard flat metrics fail to distinguish between clinically minor and severe errors. This work investigates how to quantify and mitigate abstraction errors by leveraging medical taxonomies. We benchmark several state-of-the-art VLMs using hierarchical metrics and introduce Catastrophic Abstraction Errors to capture cross-branch mistakes. Our results reveal substantial misalignment of VLMs with clinical taxonomies despite high flat performance. To address this, we propose risk-constrained thresholding and taxonomy-aware fine-tuning with radial embeddings, which reduce severe abstraction errors to below 2 per cent while maintaining competitive performance. These findings highlight the importance of hierarchical evaluation and representation-level alignment for safer and more clinically meaningful deployment of VLMs.
Paper Structure (10 sections, 2 figures, 3 tables)

This paper contains 10 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Benchmarking Setup
  4. Quantifying Catastrophic Abstraction Errors
  5. Threshold Selection for Zero-Shot classification
  6. Taxonomy-Aware Fine-Tuning
  7. Experiments and results
  8. Taxonomy-Aware Fine-Tuning
  9. Representation Alignment with Taxonomy
  10. Conclusion

Figures (2)

  • Figure 1: Illustration of a Catastrophic Abstraction Error (CAE). A chest X-ray with ground truth label fracture (FN, blue) is misclassified as nodule (FP, orange). Since the prediction and ground truth lie in disjoint branches of the medical taxonomy (top), this cross-branch error constitutes a CAE.
  • Figure 2: Three classification examples illustrating the complementary behaviour of taxonomy-aware metrics (full taxonomy omitted). : true positives; : false positives; : false negatives. Vertical bars separate taxonomic branches: thick bars for main branches, thin bars for sub-branches.