Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Strategies for Synthesizing Clinical Notes for Medical Multimodal AI

Niccolo Marini, Zhaohui Liang, Sivaramakrishnan Rajaraman, Zhiyun Xue, Sameer Antani

TL;DR

This work tackles the scarcity of large, paired multimodal biomedical data by evaluating synthetic textual clinical notes generated via LLMs to pair with dermatology images. It proposes a two-branch multimodal architecture (DenseNet121 for images, PubMedBERT for text) trained with multiple alignment losses to learn a shared representation and enable cross-modal retrieval. Through four note-generation strategies (M, P1, P2, P3), the study shows metadata-guided prompts improve cross-modal retrieval and robustness to domain shifts, while unconstrained prompts risk hallucinations that degrade alignment. The findings demonstrate the practical viability of synthetic notes for expanding multimodal dermatology AI, with implications for improved classification and retrieval under distribution shifts and for case-based retrieval tasks in clinical settings.

Abstract

Multimodal (MM) learning is emerging as a promising paradigm in biomedical artificial intelligence (AI) applications, integrating complementary modality, which highlight different aspects of patient health. The scarcity of large heterogeneous biomedical MM data has restrained the development of robust models for medical AI applications. In the dermatology domain, for instance, skin lesion datasets typically include only images linked to minimal metadata describing the condition, thereby limiting the benefits of MM data integration for reliable and generalizable predictions. Recent advances in Large Language Models (LLMs) enable the synthesis of textual description of image findings, potentially allowing the combination of image and text representations. However, LLMs are not specifically trained for use in the medical domain, and their naive inclusion has raised concerns about the risk of hallucinations in clinically relevant contexts. This work investigates strategies for generating synthetic textual clinical notes, in terms of prompt design and medical metadata inclusion, and evaluates their impact on MM architectures toward enhancing performance in classification and cross-modal retrieval tasks. Experiments across several heterogeneous dermatology datasets demonstrate that synthetic clinical notes not only enhance classification performance, particularly under domain shift, but also unlock cross-modal retrieval capabilities, a downstream task that is not explicitly optimized during training.

Evaluating Strategies for Synthesizing Clinical Notes for Medical Multimodal AI

TL;DR

This work tackles the scarcity of large, paired multimodal biomedical data by evaluating synthetic textual clinical notes generated via LLMs to pair with dermatology images. It proposes a two-branch multimodal architecture (DenseNet121 for images, PubMedBERT for text) trained with multiple alignment losses to learn a shared representation and enable cross-modal retrieval. Through four note-generation strategies (M, P1, P2, P3), the study shows metadata-guided prompts improve cross-modal retrieval and robustness to domain shifts, while unconstrained prompts risk hallucinations that degrade alignment. The findings demonstrate the practical viability of synthetic notes for expanding multimodal dermatology AI, with implications for improved classification and retrieval under distribution shifts and for case-based retrieval tasks in clinical settings.

Abstract

Multimodal (MM) learning is emerging as a promising paradigm in biomedical artificial intelligence (AI) applications, integrating complementary modality, which highlight different aspects of patient health. The scarcity of large heterogeneous biomedical MM data has restrained the development of robust models for medical AI applications. In the dermatology domain, for instance, skin lesion datasets typically include only images linked to minimal metadata describing the condition, thereby limiting the benefits of MM data integration for reliable and generalizable predictions. Recent advances in Large Language Models (LLMs) enable the synthesis of textual description of image findings, potentially allowing the combination of image and text representations. However, LLMs are not specifically trained for use in the medical domain, and their naive inclusion has raised concerns about the risk of hallucinations in clinically relevant contexts. This work investigates strategies for generating synthetic textual clinical notes, in terms of prompt design and medical metadata inclusion, and evaluates their impact on MM architectures toward enhancing performance in classification and cross-modal retrieval tasks. Experiments across several heterogeneous dermatology datasets demonstrate that synthetic clinical notes not only enhance classification performance, particularly under domain shift, but also unlock cross-modal retrieval capabilities, a downstream task that is not explicitly optimized during training.

Paper Structure

This paper contains 9 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Multimodal Architecture
  4. Datasets
  5. Clinical Notes Generation
  6. Data pre-processing
  7. Experimental Setup
  8. Results
  9. Discussion & Conclusion

Figures (1)

  • Figure 1: An overview of the proposed MM framework. Skin lesion images and associated textual clinical notes are processed by modality‑specific encoders and projected into a shared representation space that feeds a common classifier. The model is trained to optimize a cross‑entropy term for each modality, together with multiple loss functions (NT_Xent, L1-Loss and Cosine similarity) to achieve modality alignment, enhancing consistent image–text embeddings. At inference time, either modality can be used on independently.