Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ultrasound Report Generation with Multimodal Large Language Models for Standardized Texts

Peixuan Ge, Tongkun Su, Faqin Lv, Baoliang Zhao, Peng Zhang, Chi Hong Wong, Liang Yao, Yu Sun, Zenan Wang, Pak Kin Wong, Ying Hu

TL;DR

This work tackles the challenge of Ultrasound report generation, hindered by image variability and lack of standardized data, by proposing a decoder-only multimodal LLM framework enhanced with fragment-based multilingual training to produce unified, bilingual (Chinese-English) reports across multiple organs. The approach unfreezes the Vision Transformer during supervised fine-tuning and uses LoRA for efficient decoder tuning, integrating a fragment-based translation pipeline to align modular report fragments with imaging data. Empirical results show consistent improvements over KMVE and other baselines across BLEU, ROUGE-L, CIDEr, and clinical keyword metrics, with notable gains in multilingual and organ-specific tasks. The method demonstrates potential for real-world clinical workflows by enabling high-quality, native-language reports and scalable multilingual deployment, while acknowledging limitations in dataset scale and language/organ coverage that guide future work.

Abstract

Ultrasound (US) report generation is a challenging task due to the variability of US images, operator dependence, and the need for standardized text. Unlike X-ray and CT, US imaging lacks consistent datasets, making automation difficult. In this study, we propose a unified framework for multi-organ and multilingual US report generation, integrating fragment-based multilingual training and leveraging the standardized nature of US reports. By aligning modular text fragments with diverse imaging data and curating a bilingual English-Chinese dataset, the method achieves consistent and clinically accurate text generation across organ sites and languages. Fine-tuning with selective unfreezing of the vision transformer (ViT) further improves text-image alignment. Compared to the previous state-of-the-art KMVE method, our approach achieves relative gains of about 2\% in BLEU scores, approximately 3\% in ROUGE-L, and about 15\% in CIDEr, while significantly reducing errors such as missing or incorrect content. By unifying multi-organ and multi-language report generation into a single, scalable framework, this work demonstrates strong potential for real-world clinical workflows.

Ultrasound Report Generation with Multimodal Large Language Models for Standardized Texts

TL;DR

This work tackles the challenge of Ultrasound report generation, hindered by image variability and lack of standardized data, by proposing a decoder-only multimodal LLM framework enhanced with fragment-based multilingual training to produce unified, bilingual (Chinese-English) reports across multiple organs. The approach unfreezes the Vision Transformer during supervised fine-tuning and uses LoRA for efficient decoder tuning, integrating a fragment-based translation pipeline to align modular report fragments with imaging data. Empirical results show consistent improvements over KMVE and other baselines across BLEU, ROUGE-L, CIDEr, and clinical keyword metrics, with notable gains in multilingual and organ-specific tasks. The method demonstrates potential for real-world clinical workflows by enabling high-quality, native-language reports and scalable multilingual deployment, while acknowledging limitations in dataset scale and language/organ coverage that guide future work.

Abstract

Ultrasound (US) report generation is a challenging task due to the variability of US images, operator dependence, and the need for standardized text. Unlike X-ray and CT, US imaging lacks consistent datasets, making automation difficult. In this study, we propose a unified framework for multi-organ and multilingual US report generation, integrating fragment-based multilingual training and leveraging the standardized nature of US reports. By aligning modular text fragments with diverse imaging data and curating a bilingual English-Chinese dataset, the method achieves consistent and clinically accurate text generation across organ sites and languages. Fine-tuning with selective unfreezing of the vision transformer (ViT) further improves text-image alignment. Compared to the previous state-of-the-art KMVE method, our approach achieves relative gains of about 2\% in BLEU scores, approximately 3\% in ROUGE-L, and about 15\% in CIDEr, while significantly reducing errors such as missing or incorrect content. By unifying multi-organ and multi-language report generation into a single, scalable framework, this work demonstrates strong potential for real-world clinical workflows.
Paper Structure (8 sections, 1 equation, 1 figure, 2 tables)

This paper contains 8 sections, 1 equation, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Decoder-Only Multimodal LLMs
  4. Dataset and Fragment-Based Multilingual Training Pipeline
  5. Experiments and Results
  6. Comparison of Traditional Methods and MLLMs
  7. Multilingual Training and Ablation Studies
  8. Discussion and Conclusion

Figures (1)

  • Figure 1: (1) The MLLM architecture. *Target report content included only during training to compute the masked causal loss. (2) Fragment distribution by total occurrences and unique counts across organ sites. (3, 4, 5) Example input ultrasound images at thyroid site, raw Chinese and translated English reports, and fragment-translation lookup table mapping Chinese fragments to English equivalents. (6) Multilingual training prompts for joint and cross-language generation. (7) Example inputs and outputs for ground truth, proposed methods and the old method.