Multimodal Task Representation Memory Bank vs. Catastrophic Forgetting in Anomaly Detection

TL;DR

The paper tackles unsupervised continual anomaly detection (UCAD) under multi-task settings by addressing catastrophic forgetting caused by incomplete multimodal representations. It introduces the Multimodal Task Representation Memory Bank (MTRMB), which combines a Key-Prompt-Multimodal Knowledge (KPMK) mechanism with Refined Structure-based Contrastive Learning (RSCL) to preserve prior task knowledge while adapting to new tasks, enabling cross-modal fusion between BERT and ViT. During inference, MTRMB retrieves task-specific prompts and uses segmentation-guided structure masks from Grounding DINO and SAM to produce compact, transferable representations and robust anomaly scores. Experiments on MVTEC AD and VisA show state-of-the-art continual detection performance with an average detection accuracy of $0.921$ at the lowest forgetting rate, and the authors plan to open-source the implementation on GitHub.

Abstract

Unsupervised Continuous Anomaly Detection (UCAD) faces significant challenges in multi-task representation learning, with existing methods suffering from incomplete representation and catastrophic forgetting. Unlike supervised models, unsupervised scenarios lack prior information, making it difficult to effectively distinguish redundant and complementary multimodal features. To address this, we propose the Multimodal Task Representation Memory Bank (MTRMB) method through two key technical innovations: A Key-Prompt-Multimodal Knowledge (KPMK) mechanism that uses concise key prompts to guide cross-modal feature interaction between BERT and ViT. Refined Structure-based Contrastive Learning (RSCL) leveraging Grounding DINO and SAM to generate precise segmentation masks, pulling features of the same structural region closer while pushing different structural regions apart. Experiments on MVtec AD and VisA datasets demonstrate MTRMB's superiority, achieving an average detection accuracy of 0.921 at the lowest forgetting rate, significantly outperforming state-of-the-art methods. We plan to open source on GitHub.

Figures (5)

• Figure 1: Comparison between UCAD with a memory bank and UCAD with a multimodal Task Representation Memory Bank. a) The memory bank-based method has the problem of forgetting old task knowledge due to incomplete representation, resulting in inaccurate new task hyperplane boundary. b) Introducing multimodal features to construct a task representation memory can effectively retain the intrinsic information of the old task and obtain a better new task hyperplane boundary.
• Figure 2: The framework of UCAD using multimodal Task Representation Memory Bank. (a) Text-image data is input during the training phase, and an effective task intrinsic memory bank is formed through the KPMK. In addition, we use RSCL to better utilize task-related contextual information to obtain a more compact MTRMB. (b) When a test image is input during the testing phase, the framework automatically queries the Task key to retrieve the corresponding task prompts, completes the model's transfer of task knowledge through the prompts, then extracts the features of the test image and calculates the similarity with normal knowledge, and finally completes continuous detection of anomalies. (c) The KPMK mechanism uses the concise key to guide the cross-fusion of features from two different modalities, text and image, and generates an effective task representation memory bank.
• Figure 3: The T-SNE Visualization results of MTRMB. The dimensionality reduction visualization results of 15 classes show that the proposed method can generate a compact task representation memory bank for UCAD. It should be noted that the arc-shaped dimensionality reduction feature sets are all generated by the background area. This is because the background areas of each category in the MVTec AD dataset are very similar.
• Figure 4: The first row shows the test images, the second row shows the structure regions generated by SAM, and the third row shows the structure regions generated by the effective interaction between only Grounding DINO and SAM. The segmented region visualization shows that our method can generate more accurate structure regions to guide the model to learn the key parts of the object.
• Figure 5: Visualization examples of continual anomaly detection. The first row displays the original anomaly images, the second row shows the ground truth annotations, and the third to fifth rows depict the heatmaps of our method and other methods.