AnyAD: Unified Any-Modality Anomaly Detection in Incomplete Multi-Sequence MRI

TL;DR

The paper tackles the challenge of detecting anomalies in brain MRI when some modalities are missing. It proposes AnyAD, a unified architecture that uses a dual-pathway DINOv2 encoder, feature distribution alignment, and an INP-guided reconstruction to learn modality-invariant normal patterns and localize anomalies across arbitrary modality configurations without retraining. Key contributions include the Any-Modality feature distribution alignment, an indirect feature-completion mechanism via random modality masking, and a prototype-based INP learning module that reduces shortcut learning, demonstrated across BraTS2018, MU-Glioma-Post, and Pretreat-MetsToBrain-Masks with strong cross-modality and cross-domain generalization. The method achieves state-of-the-art performance on seven modality combos, exhibits zero-shot capabilities on unseen modalities, and provides a scalable, clinically relevant framework for multimodal medical AD under imperfect acquisition conditions; code is publicly available.

Abstract

Reliable anomaly detection in brain MRI remains challenging due to the scarcity of annotated abnormal cases and the frequent absence of key imaging modalities in real clinical workflows. Existing single-class or multi-class anomaly detection (AD) models typically rely on fixed modality configurations, require repetitive training, or fail to generalize to unseen modality combinations, limiting their clinical scalability. In this work, we present a unified Any-Modality AD framework that performs robust anomaly detection and localization under arbitrary MRI modality availability. The framework integrates a dual-pathway DINOv2 encoder with a feature distribution alignment mechanism that statistically aligns incomplete-modality features with full-modality representations, enabling stable inference even with severe modality dropout. To further enhance semantic consistency, we introduce an Intrinsic Normal Prototypes (INPs) extractor and an INP-guided decoder that reconstruct only normal anatomical patterns while naturally amplifying abnormal deviations. Through randomized modality masking and indirect feature completion during training, the model learns to adapt to all modality configurations without re-training. Extensive experiments on BraTS2018, MU-Glioma-Post, and Pretreat-MetsToBrain-Masks demonstrate that our approach consistently surpasses state-of-the-art industrial and medical AD baselines across 7 modality combinations, achieving superior generalization. This study establishes a scalable paradigm for multimodal medical AD under real-world, imperfect modality conditions. Our source code is available at https://github.com/wuchangw/AnyAD.

Figures (8)

• Figure 1: Overview of our AnyAD framework. (a) Our model consists of a pre-trained encoder, an INPs extractor, a bottleneck, and an INPs-guided decoder. Simultaneously, the dual paths extract the means and variances of the full-modal features and the missing-modal features, respectively. (b) Detailed process of INPs extractor and INPs consistency loss. (c) Detailed architecture of the INPs-guided decoder. (d)Detailed process of feature distribution alignment loss.
• Figure 2: Anomaly localization visualizations of the top-5 models on the BraTS2018 dataset. Each row corresponds to one modality combination, numbered from 1 to 7, with detailed compositions listed in the first three columns. This convention for referring to modality combinations is consistent throughout the paper. From left to right, the columns show: the modality combination details, the ground truth segmentation mask as reference standard, and alternating rows of anomaly heatmaps and model-predicted localization maps, where odd rows display heatmaps and even rows show prediction maps.
• Figure 3: Anomaly localization visualizations of the top-5 models on the MU-Glioma-Post dataset. Each row corresponds to one modality combination, numbered from 1 to 7, with detailed compositions listed in the first three columns. This convention for referring to modality combinations is consistent throughout the paper. From left to right, the columns show: the modality combination details, the ground truth segmentation mask as reference standard, and alternating rows of anomaly heatmaps and model-predicted localization maps, where odd rows display heatmaps and even rows show prediction maps.
• Figure 4: Threshold curves and confusion matrices of Any-AD on BraTS2018 (top two rows) and MU-Glioma-Post (bottom two rows) datasets. From left to right, the columns correspond to modality combinations 1-7.
• Figure 5: Distribution and low-dimensional representation of anomaly scores. Top row: histograms of anomaly scores; bottom row: t-SNE visualization of the scores.