Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QwenCLIP: Boosting Medical Vision-Language Pretraining via LLM Embeddings and Prompt tuning

Xiaoyang Wei, Camille Kurtz, Florence Cloppet

TL;DR

Medical CLIP models are limited by a 77-token text encoder, truncating long radiology reports and hindering image-text alignment. QwenCLIP replaces the CLIP text encoder with the Qwen3-Embedding LLM and introduces a hybrid prompt-tuning strategy to generate long, semantically rich text embeddings that align with visual features via a contrastive objective. The approach delivers state-of-the-art zero-shot image retrieval on ROCOv2 and IRMA, outperforming domain-adapted BERT variants and recent LLM-based CLIP methods, and demonstrating the value of LLM embeddings for medical vision-language modeling. By enabling long-form report understanding and scalable cross-modal alignment, QwenCLIP offers a practical route toward richer medical multimodal representation with available code for reproducibility.

Abstract

Contrastive Language-Image Pretraining (CLIP) has demonstrated strong generalization for vision-language tasks in computer vision and medical domains, yet its text encoder accepts only up to 77 tokens, which limits its ability to represent long and information-rich radiology reports. Recent adaptations using domain-specific encoders, such as PubMedBERT or ClinicalBERT, mitigate this issue by leveraging medical corpora, but remain constrained by their limited input length (typically 512 tokens) and relatively shallow semantic understanding. To address these limitations, we propose QwenCLIP, a vision-language framework that replaces CLIP's text encoder with a large language model (LLM)-based embedding module (e.g., Qwen3-Embedding) and introduces learnable prompts to enhance cross-modal alignment. By leveraging the extended context window and richer representations of LLMs, QwenCLIP captures comprehensive medical semantics from long-form clinical text, substantially improving medical image-text alignment and downstream performance on radiology benchmarks. Our code is publicly available at https://github.com/Wxy-24/QwenCLIP.

QwenCLIP: Boosting Medical Vision-Language Pretraining via LLM Embeddings and Prompt tuning

TL;DR

Medical CLIP models are limited by a 77-token text encoder, truncating long radiology reports and hindering image-text alignment. QwenCLIP replaces the CLIP text encoder with the Qwen3-Embedding LLM and introduces a hybrid prompt-tuning strategy to generate long, semantically rich text embeddings that align with visual features via a contrastive objective. The approach delivers state-of-the-art zero-shot image retrieval on ROCOv2 and IRMA, outperforming domain-adapted BERT variants and recent LLM-based CLIP methods, and demonstrating the value of LLM embeddings for medical vision-language modeling. By enabling long-form report understanding and scalable cross-modal alignment, QwenCLIP offers a practical route toward richer medical multimodal representation with available code for reproducibility.

Abstract

Contrastive Language-Image Pretraining (CLIP) has demonstrated strong generalization for vision-language tasks in computer vision and medical domains, yet its text encoder accepts only up to 77 tokens, which limits its ability to represent long and information-rich radiology reports. Recent adaptations using domain-specific encoders, such as PubMedBERT or ClinicalBERT, mitigate this issue by leveraging medical corpora, but remain constrained by their limited input length (typically 512 tokens) and relatively shallow semantic understanding. To address these limitations, we propose QwenCLIP, a vision-language framework that replaces CLIP's text encoder with a large language model (LLM)-based embedding module (e.g., Qwen3-Embedding) and introduces learnable prompts to enhance cross-modal alignment. By leveraging the extended context window and richer representations of LLMs, QwenCLIP captures comprehensive medical semantics from long-form clinical text, substantially improving medical image-text alignment and downstream performance on radiology benchmarks. Our code is publicly available at https://github.com/Wxy-24/QwenCLIP.

Paper Structure

This paper contains 13 sections, 6 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed framework: QwenCLIP
  3. Hybrid prompt tuning for LLM-based text encoder
  4. Vision-language contrastive learning
  5. Experimental study
  6. Datasets
  7. Implementation details
  8. Downstream tasks
  9. Results
  10. Ablation studies
  11. Conclusion
  12. Compliance with ethical standards
  13. Acknowledgments

Figures (2)

  • Figure 1: Text length distribution for common medical image-text pair datasets.
  • Figure 2: Overview of the QwenCLIP framework, where instruction-tuned soft prompts guide the frozen LLM and learnable image encoder to align multimodal representations