EvalBlocks: A Modular Pipeline for Rapidly Evaluating Foundation Models in Medical Imaging

TL;DR

The paper tackles slow and error-prone evaluation workflows during foundation-model development in medical imaging. It introduces EvalBlocks, a modular, Snakemake-based pipeline that supports plug-and-play blocks for feature extraction, aggregation, and evaluation, with centralized tracking and caching for reproducible, scalable experimentation. The authors demonstrate the approach by evaluating five foundation models across three patch-level malignancy tasks, illustrating rapid prototyping of aggregation methods and evaluation strategies and showing tangible reductions in compute time through caching and parallelism. This framework lowers evaluation logistics overhead, enabling researchers to focus on architectural and training innovations, and sets the stage for broader integration and future extensions to other task types and platforms.

Abstract

Developing foundation models in medical imaging requires continuous monitoring of downstream performance. Researchers are burdened with tracking numerous experiments, design choices, and their effects on performance, often relying on ad-hoc, manual workflows that are inherently slow and error-prone. We introduce EvalBlocks, a modular, plug-and-play framework for efficient evaluation of foundation models during development. Built on Snakemake, EvalBlocks supports seamless integration of new datasets, foundation models, aggregation methods, and evaluation strategies. All experiments and results are tracked centrally and are reproducible with a single command, while efficient caching and parallel execution enable scalable use on shared compute infrastructure. Demonstrated on five state-of-the-art foundation models and three medical imaging classification tasks, EvalBlocks streamlines model evaluation, enabling researchers to iterate faster and focus on model innovation rather than evaluation logistics. The framework is released as open source software at https://github.com/DIAGNijmegen/eval-blocks.

Figures (4)

• Figure 1.1: In our framework, pipeline steps are implemented as self-contained blocks. Foundation models embed input patches, and these feature embeddings can be optionally aggregated and then evaluated. The pipeline blocks can be freely extended and plugged into each other, enabling fast, reproducible, and customizable evaluation during foundation model development.
• Figure 1.2: A visualization of model results on our CT datasets created by running EvalBlocks, with error bars depicting the standard deviation across folds. While CT-FM 3865-09 and Curia 3865-10 perform best on PANORAMA 3865-07, UMedPT 3865-11 is slightly more accurate on AMARA. Our pipeline allows for fast and automated comparison between models and checkpoints.
• Figure 1.3: EvalBlocks also enables evaluation across modalities and aggregation methods, here for the PI-CAI 3865-08 dataset. Error bars denote the standard deviation across folds. Overall, ADC is the most informative modality for malignancy discrimination. The modality mean aggregation emerges as a well-performing strategy for this task. Our framework enables researchers to easily prototype aggregation methods.
• Figure 1.4: Visualizations of the feature embeddings of Curia 3865-10 on the first fold of the PANORAMA dataset 3865-07. While PCA and t-SNE yield no clusters, LDA shows two distinct peaks for the two classes. This reveals that the model produces linearly separable feature embeddings for this task in label-dependent directions, but not in directions of maximum variance or local neighborhood structures. EvalBlocks produces these visualizations for all folds, datasets, and models, allowing deeper analysis where necessary.