Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languages

Marco Salmè, Rosa Sicilia, Paolo Soda, Valerio Guarrasi

TL;DR

The paper tackles radiology report generation in multilingual, low-resource contexts by benchmarking instruction-tuned Vision-Language Models. Using the LLaVA framework with a frozen MedSAM encoder, a multimodal adapter, and LoRA-tuned decoders, it compares general English pretraining, English medical fine-tuning, and language-specific fine-tuning for Italian, German, and Spanish on the IU-Xray dataset (augmented with translations). It finds that language-specific tuning consistently outperforms general and medical baselines, with Spanish achieving the strongest results, and that low-temperature settings favor reliability in language-specific generations. The work highlights the critical role of language adaptation for multilingual clinical NLP and provides guidance on tuning strategies, while outlining limitations and avenues for future research in multilingual radiology report generation.

Abstract

The integration of artificial intelligence in healthcare has opened new horizons for improving medical diagnostics and patient care. However, challenges persist in developing systems capable of generating accurate and contextually relevant radiology reports, particularly in low-resource languages. In this study, we present a comprehensive benchmark to evaluate the performance of instruction-tuned Vision-Language Models (VLMs) in the specialized task of radiology report generation across three low-resource languages: Italian, German, and Spanish. Employing the LLaVA architectural framework, we conducted a systematic evaluation of pre-trained models utilizing general datasets, domain-specific datasets, and low-resource language-specific datasets. In light of the unavailability of models that possess prior knowledge of both the medical domain and low-resource languages, we analyzed various adaptations to determine the most effective approach for these contexts. The results revealed that language-specific models substantially outperformed both general and domain-specific models in generating radiology reports, emphasizing the critical role of linguistic adaptation. Additionally, models fine-tuned with medical terminology exhibited enhanced performance across all languages compared to models with generic knowledge, highlighting the importance of domain-specific training. We also explored the influence of the temperature parameter on the coherence of report generation, providing insights for optimal model settings. Our findings highlight the importance of tailored language and domain-specific training for improving the quality and accuracy of radiological reports in multilingual settings. This research not only advances our understanding of VLMs adaptability in healthcare but also points to significant avenues for future investigations into model tuning and language-specific adaptations.

Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languages

TL;DR

The paper tackles radiology report generation in multilingual, low-resource contexts by benchmarking instruction-tuned Vision-Language Models. Using the LLaVA framework with a frozen MedSAM encoder, a multimodal adapter, and LoRA-tuned decoders, it compares general English pretraining, English medical fine-tuning, and language-specific fine-tuning for Italian, German, and Spanish on the IU-Xray dataset (augmented with translations). It finds that language-specific tuning consistently outperforms general and medical baselines, with Spanish achieving the strongest results, and that low-temperature settings favor reliability in language-specific generations. The work highlights the critical role of language adaptation for multilingual clinical NLP and provides guidance on tuning strategies, while outlining limitations and avenues for future research in multilingual radiology report generation.

Abstract

The integration of artificial intelligence in healthcare has opened new horizons for improving medical diagnostics and patient care. However, challenges persist in developing systems capable of generating accurate and contextually relevant radiology reports, particularly in low-resource languages. In this study, we present a comprehensive benchmark to evaluate the performance of instruction-tuned Vision-Language Models (VLMs) in the specialized task of radiology report generation across three low-resource languages: Italian, German, and Spanish. Employing the LLaVA architectural framework, we conducted a systematic evaluation of pre-trained models utilizing general datasets, domain-specific datasets, and low-resource language-specific datasets. In light of the unavailability of models that possess prior knowledge of both the medical domain and low-resource languages, we analyzed various adaptations to determine the most effective approach for these contexts. The results revealed that language-specific models substantially outperformed both general and domain-specific models in generating radiology reports, emphasizing the critical role of linguistic adaptation. Additionally, models fine-tuned with medical terminology exhibited enhanced performance across all languages compared to models with generic knowledge, highlighting the importance of domain-specific training. We also explored the influence of the temperature parameter on the coherence of report generation, providing insights for optimal model settings. Our findings highlight the importance of tailored language and domain-specific training for improving the quality and accuracy of radiological reports in multilingual settings. This research not only advances our understanding of VLMs adaptability in healthcare but also points to significant avenues for future investigations into model tuning and language-specific adaptations.
Paper Structure (10 sections, 4 figures, 1 table)

This paper contains 10 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Experimental Setup
  4. Materials
  5. Training details
  6. Evalutation Metrics
  7. Results and Discussion
  8. Comparative Evaluation of LLMs
  9. Temperature Analysis
  10. Conclusions

Figures (4)

  • Figure 1: The used architectural framework.The instruction-tuned VLM generates radiology reports based on a fixed input prompt: "Provide the findings of the following radiology image.".
  • Figure 2: Schematic representation of the methodological approach.
  • Figure 3: Comparative Histograms of Evaluation Metrics for RRG Models. The top row displays results for the BLEU metric across different models, while the second row presents histograms for the ROUGE and METEOR metrics.
  • Figure 4: Temperature Analysis.