Zero-Training Task-Specific Model Synthesis for Few-Shot Medical Image Classification

TL;DR

This work addresses the critical data scarcity problem in medical image analysis by proposing Zero-Training Task-Specific Model Synthesis (ZS-TMS), which directly generates the entire set of classifier weights from minimal multi-modal input (e.g., a single image and a clinical text description) using the Semantic-Guided Parameter Synthesizer (SGPS). SGPS employs multi-modal encoders (ViT for images and ClinicalBERT for text) and a Transformer-based generator to output a ready-to-use, lightweight classifier without task-specific training. Across ISIC-FS and RareDerm-FS benchmarks, SGPS achieves state-of-the-art performance, with pronounced gains in 1-shot scenarios, validating the potential to rapidly develop AI tools for the long tail of rare diseases. The work highlights practical impact by enabling deployment of diagnostic models under extreme data constraints, while also noting computational costs, dependence on description quality, and opportunities for extending to denser tasks and more efficient architectures. Overall, SGPS represents a paradigm shift from learning-to-adapt to task-level generation, offering a data-efficient path to clinically useful AI systems.

Abstract

Deep learning models have achieved remarkable success in medical image analysis but are fundamentally constrained by the requirement for large-scale, meticulously annotated datasets. This dependency on "big data" is a critical bottleneck in the medical domain, where patient data is inherently difficult to acquire and expert annotation is expensive, particularly for rare diseases where samples are scarce by definition. To overcome this fundamental challenge, we propose a novel paradigm: Zero-Training Task-Specific Model Synthesis (ZS-TMS). Instead of adapting a pre-existing model or training a new one, our approach leverages a large-scale, pre-trained generative engine to directly synthesize the entire set of parameters for a task-specific classifier. Our framework, the Semantic-Guided Parameter Synthesizer (SGPS), takes as input minimal, multi-modal task information as little as a single example image (1-shot) and a corresponding clinical text description to directly synthesize the entire set of parameters for a task-specific classifier. The generative engine interprets these inputs to generate the weights for a lightweight, efficient classifier (e.g., an EfficientNet-V2), which can be deployed for inference immediately without any task-specific training or fine-tuning. We conduct extensive evaluations on challenging few-shot classification benchmarks derived from the ISIC 2018 skin lesion dataset and a custom rare disease dataset. Our results demonstrate that SGPS establishes a new state-of-the-art, significantly outperforming advanced few-shot and zero-shot learning methods, especially in the ultra-low data regimes of 1-shot and 5-shot classification. This work paves the way for the rapid development and deployment of AI-powered diagnostic tools, particularly for the long tail of rare diseases where data is critically limited.

Figures (10)

• Figure 1: The Challenge of Data Scarcity in Medical AI
• Figure 2: A Taxonomy of Learning Paradigms for Data Scarcity
• Figure 3: A Taxonomy of Learning Paradigms for Data Scarcity
• Figure 4: Image Encoding Process
• Figure 5: Text Encoding Process