UMIT: Unifying Medical Imaging Tasks via Vision-Language Models

TL;DR

UMIT tackles the fragmentation of medical image analysis by unifying diverse tasks and modalities under a single vision-language framework. It employs a two-stage training pipeline—feature alignment to fuse visual and textual representations, followed by instruction-tuning with task-aware prompts—to enable cross-task transfer and bilingual (English/Chinese) support across $2D$ and $3D$ medical data. Across five tasks and $18$ public datasets, UMIT achieves state-of-the-art or competitive results on medical VQA, classification, report generation, disease detection, and landmark detection, demonstrating strong generalization and cross-modal capabilities. The approach promises meaningful clinical impact by improving diagnostic efficiency and enabling multilingual, multi-task medical AI in diverse healthcare settings.

Abstract

With the rapid advancement of deep learning, particularly in the field of medical image analysis, an increasing number of Vision-Language Models (VLMs) are being widely applied to solve complex health and biomedical challenges. However, existing research has primarily focused on specific tasks or single modalities, which limits their applicability and generalization across diverse medical scenarios. To address this challenge, we propose UMIT, a unified multi-modal, multi-task VLM designed specifically for medical imaging tasks. UMIT is able to solve various tasks, including visual question answering, disease detection, and medical report generation. In addition, it is applicable to multiple imaging modalities (e.g., X-ray, CT and PET), covering a wide range of applications from basic diagnostics to complex lesion analysis. Moreover, UMIT supports both English and Chinese, expanding its applicability globally and ensuring accessibility to healthcare services in different linguistic contexts. To enhance the model's adaptability and task-handling capability, we design a unique two-stage training strategy and fine-tune UMIT with designed instruction templates. Through extensive empirical evaluation, UMIT outperforms previous methods in five tasks across multiple datasets. The performance of UMIT indicates that it can significantly enhance diagnostic accuracy and workflow efficiency, thus providing effective solutions for medical imaging applications.

Figures (5)

• Figure 1: Performance comparison with various models. UMIT utilizes all the datasets and demonstrates broad applicability across multiple tasks and modalities.
• Figure 2: The framework of the proposed UMIT. (a) Overview of our two-stage training strategy. (b) Training tasks in feature alignment stage. (c) Training tasks in instruction-tuning stage.
• Figure 3: Visualization of UMIT's results across five tasks.
• Figure 4: Qualitative examples of Medical VQA task.
• Figure 5: Qualitative examples of classification task, report generation task, disease detection task and landmark detection task.