SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learning

TL;DR

The paper addresses evaluating multimodal in-context learning for medical tasks using expert-designed demonstrations. It proposes SMMILE and SMMILE++ as benchmarks with 111 problems (517 Q-A triplets) and 1038 permuted problems, spanning 6 specialties and 13 imaging modalities. Across 15 MLLMs, results reveal limited ICL gains, strong model- and task-specific variability, and sensitivity to in-context quality and ordering. The work provides a rigorous evaluation framework and analysis toolkit to guide future development of reliable medical multimodal ICL systems.

Abstract

Multimodal in-context learning (ICL) remains underexplored despite significant potential for domains such as medicine. Clinicians routinely encounter diverse, specialized tasks requiring adaptation from limited examples, such as drawing insights from a few relevant prior cases or considering a constrained set of differential diagnoses. While multimodal large language models (MLLMs) have shown advances in medical visual question answering (VQA), their ability to learn multimodal tasks from context is largely unknown. We introduce SMMILE, the first expert-driven multimodal ICL benchmark for medical tasks. Eleven medical experts curated problems, each including a multimodal query and multimodal in-context examples as task demonstrations. SMMILE encompasses 111 problems (517 question-image-answer triplets) covering 6 medical specialties and 13 imaging modalities. We further introduce SMMILE++, an augmented variant with 1038 permuted problems. A comprehensive evaluation of 15 MLLMs demonstrates that most models exhibit moderate to poor multimodal ICL ability in medical tasks. In open-ended evaluations, ICL contributes only an 8% average improvement over zero-shot on SMMILE and 9.4% on SMMILE++. We observe a susceptibility for irrelevant in-context examples: even a single noisy or irrelevant example can degrade performance by up to 9.5%. Moreover, we observe that MLLMs are affected by a recency bias, where placing the most relevant example last can lead to substantial performance improvements of up to 71%. Our findings highlight critical limitations and biases in current MLLMs when learning multimodal medical tasks from context. SMMILE is available at https://smmile-benchmark.github.io.

Figures (13)

• Figure 1: Overview of the SMMILE benchmark. In order to test the ability of MLLMs to perform multimodal in-context learning in the medical domain, we curate an expert-annotated dataset consisting of multimodal queries paired with two or more task-specific in-context examples. In contrast to prior few-shot evaluations, our in-context examples are expert-designed demonstrations of the task at hand, rather than randomly retrieved examples.
• Figure 2: Web interface for data collection.
• Figure 3: Dataset characteristics. (A–D) Distribution of four key categorical annotations across the unique problems: (A) answer format, (B) rarity of the clinical case based on how often clinicians would experience the medical concepts included in each problem, (C) primary cognitive process required (where reasoning classification is defined by final problem not having direct support in its in-context example set), and (D) rated difficulty. (E–F) Horizontal barplots showing the breakdown of each problem by its main medical specialty (E) and by main image type used (F). (G) Histogram of the number of in-context examples provided per problem. (H) Overlaid histograms of the character-length distributions for questions versus answers. All panels are based on the 111 problems included in SMMILE.
• Figure 4: We provide a fine-grained breakdown of MLLM performance on the SMMILE benchmark. We report performance stratified by answer format (Panel A), cognitive process necessary to obtain the answer (Panel B), number of in-context examples provided to the model (Panel C), and image type (Panel D). Here, we focus on open-ended evaluations, and the y-axis represents prediction accuracy as computed by the LLM-as-a-Judge approach. The acronym MG refers to Mammograms.
• Figure 5: We analyze the effect of example order on MLLM performance. We report performance across 9 MLLMs (ordered by model size) in the open-ended setting with LLM-as-a-Judge evaluation.