Robust Modality-incomplete Anomaly Detection: A Modality-instructive Framework with Benchmark

TL;DR

This work tackles the practical challenge of industrial anomaly detection when multimodal data are incomplete. It formalizes Modality-Incomplete Industrial Anomaly Detection (MIIAD) and introduces the MIIAD Bench to benchmark robustness under missing modalities, then proposes RADAR, a two-stage framework that combines Adaptive Instruction Fusion with a HyperNetwork and a Double-Pseudo Hybrid Detection scheme, plus repository-based scoring via Mahalanobis distance and OCSVM. RADAR fuses available modalities through modality-incomplete instructions $I_m$ and dynamic parameters, while enforcing global reconstruction and local contrastive supervision to mitigate overfitting, achieving superior performance across missing-rate settings. The results demonstrate strong robustness and parameter efficiency, suggesting practical impact for real-world industrial inspection where modality availability is often imperfect ($\\eta\%$ missing rates such as $30\%$, $50\%$, and $70\%$).

Abstract

Multimodal Industrial Anomaly Detection (MIAD), which utilizes 3D point clouds and 2D RGB images to identify abnormal regions in products, plays a crucial role in industrial quality inspection. However, traditional MIAD settings assume that all 2D and 3D modalities are paired, ignoring the fact that multimodal data collected from the real world is often imperfect due to missing modalities. Additionally, models trained on modality-incomplete data are prone to overfitting. Therefore, MIAD models that demonstrate robustness against modality-incomplete data are highly desirable in practice. To address this, we introduce a pioneering study that comprehensively investigates Modality-Incomplete Industrial Anomaly Detection (MIIAD), and under the guidance of experts, we construct the MIIAD Bench with rich modality-missing settings to account for imperfect learning environments with incomplete multimodal information. As expected, we find that most existing MIAD methods perform poorly on the MIIAD Bench, leading to significant performance degradation. To tackle this challenge, we propose a novel two-stage Robust modAlity-aware fusing and Detecting framewoRk, abbreviated as RADAR. Specifically: i) We propose Modality-incomplete Instruction to guide the multimodal Transformer to robustly adapt to various modality-incomplete scenarios, and implement adaptive parameter learning based on HyperNetwork. ii) Then, we construct a Double-Pseudo Hybrid Module to highlight the uniqueness of modality combinations, mitigating overfitting issues and further enhancing the robustness of the MIIAD model. Our experimental results demonstrate that the proposed RADAR significantly outperforms traditional MIAD methods on our newly created MIIAD dataset, proving its practical application value.

Figures (8)

• Figure 1: (a). Outline of MIIAD. MIIAD processes modallity-incomplete data, generating anomaly scores and segmentation maps. Our model RADAR surpasses most MIAD methods on this task. (b). Overfitting from modality-incomplete training. Training with incomplete-modal data can cause the model to focus on irrelevant features, leading to overfitting.
• Figure 2: Overview of RADAR. It consists of three parts: i) MIIAD Bench Data Collection. Collecting data from the MVTec-3D AD dataset and constructing different data splits for the benchmark under the guidance of domain experts. ii) Adaptive Instruction Fusion. Prepending the modality-incomplete instruction $I_m$ of multimodal data $(x^{m_1}_i, x^{m_2}_i)$ into a simple multimodal Transformer input token, while introducing HyperNetwork to achieve adaptive parameter learning. iii) Double-Pseudo Hybrid Detection. Constructing a Double-Pseudo Hybrid Module for mitigating overfitting and feeding the features stored in multiple repositories $R_{rgb}$, $R_{fs}$, and $R_{pc}$ into the MDM and OCSVM for anomaly score $\tau$ and segmentation map, respectively.
• Figure 3: Quantitative results of I-AUROC on the MIIAD Bench - MVTEC-3D AD with different missing rates under different missing-modality scenarios. Each data point in the figure represents that training and testing are with the same $\eta\%$ missing rate.
• Figure 4: Effect of the position of instruction layers.
• Figure 5: Visualization results of IAD.