Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection

Chengjie Wang, Wenbing Zhu, Bin-Bin Gao, Zhenye Gan, Jianning Zhang, Zhihao Gu, Shuguang Qian, Mingang Chen, Lizhuang Ma

TL;DR

A large-scale, Real-world, and multi-view Industrial Anomaly Detection dataset, named Real- I AD, which contains 150K high-resolution images of 30 different objects, an order of magnitude larger than existing datasets, making it more challenging than previous datasets.

Abstract

Industrial anomaly detection (IAD) has garnered significant attention and experienced rapid development. However, the recent development of IAD approach has encountered certain difficulties due to dataset limitations. On the one hand, most of the state-of-the-art methods have achieved saturation (over 99% in AUROC) on mainstream datasets such as MVTec, and the differences of methods cannot be well distinguished, leading to a significant gap between public datasets and actual application scenarios. On the other hand, the research on various new practical anomaly detection settings is limited by the scale of the dataset, posing a risk of overfitting in evaluation results. Therefore, we propose a large-scale, Real-world, and multi-view Industrial Anomaly Detection dataset, named Real-IAD, which contains 150K high-resolution images of 30 different objects, an order of magnitude larger than existing datasets. It has a larger range of defect area and ratio proportions, making it more challenging than previous datasets. To make the dataset closer to real application scenarios, we adopted a multi-view shooting method and proposed sample-level evaluation metrics. In addition, beyond the general unsupervised anomaly detection setting, we propose a new setting for Fully Unsupervised Industrial Anomaly Detection (FUIAD) based on the observation that the yield rate in industrial production is usually greater than 60%, which has more practical application value. Finally, we report the results of popular IAD methods on the Real-IAD dataset, providing a highly challenging benchmark to promote the development of the IAD field.

Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection

TL;DR

A large-scale, Real-world, and multi-view Industrial Anomaly Detection dataset, named Real- I AD, which contains 150K high-resolution images of 30 different objects, an order of magnitude larger than existing datasets, making it more challenging than previous datasets.

Abstract

Industrial anomaly detection (IAD) has garnered significant attention and experienced rapid development. However, the recent development of IAD approach has encountered certain difficulties due to dataset limitations. On the one hand, most of the state-of-the-art methods have achieved saturation (over 99% in AUROC) on mainstream datasets such as MVTec, and the differences of methods cannot be well distinguished, leading to a significant gap between public datasets and actual application scenarios. On the other hand, the research on various new practical anomaly detection settings is limited by the scale of the dataset, posing a risk of overfitting in evaluation results. Therefore, we propose a large-scale, Real-world, and multi-view Industrial Anomaly Detection dataset, named Real-IAD, which contains 150K high-resolution images of 30 different objects, an order of magnitude larger than existing datasets. It has a larger range of defect area and ratio proportions, making it more challenging than previous datasets. To make the dataset closer to real application scenarios, we adopted a multi-view shooting method and proposed sample-level evaluation metrics. In addition, beyond the general unsupervised anomaly detection setting, we propose a new setting for Fully Unsupervised Industrial Anomaly Detection (FUIAD) based on the observation that the yield rate in industrial production is usually greater than 60%, which has more practical application value. Finally, we report the results of popular IAD methods on the Real-IAD dataset, providing a highly challenging benchmark to promote the development of the IAD field.
Paper Structure (17 sections, 5 figures, 10 tables)

This paper contains 17 sections, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Anomaly Detection Datasets
  4. Standard Anomaly Detection
  5. Other Settings in Anomaly Detection
  6. Real-IAD Dataset Description
  7. Data Collection and Construction Manner
  8. Comparison with Popular 2D Datasets
  9. Real-IAD Visualization
  10. Benchmark
  11. Evalution settings
  12. Evaluation Metric
  13. Comparisons with IAD Benchmarks
  14. Results on Unsupervised IAD
  15. IAD Benchmarks Meet FUIAD
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Data collection pipeline for our proposed Real-IAD dataset, which consists of four steps in tandem: (a) Material preparation and defect manufacturing. (b) Prototype design and construction that contains 5 shots for capturing multi-view images simultaneously, i.e., one top-down camera with extra four cameras arranged uniformly at 45 degrees. (c) Collection procedure that includes cyclical processes of data collection, annotation, and cleaning. At the bottom, partial visualization of the final Real-IAD dataset reveals that the proposed Real-IAD exhibits large scale (30 classes), a wide range of defect proportions (0.01% to 6.75%), and a broad defect ratio (1:1 to 1:10), indicating that Real-IAD is highly challenging. Abnormal areas are prominently marked in red.
  • Figure 2: Statistic information of our proposed Real-IAD datast.(a) Statistic comparison of anomaly/normal data volume among popular datasets. (b) Statistics of the percentage of the image area occupied by the anomaly region. (c) Statistics of the aspect ratio of the minimum bounding rectangle of the defect. (d) Distribution of anomaly/normal image quantities across different categories. (e) Distribution of data volume across different defect categories.
  • Figure 3: Visualization of multi-view collections, where the first column represents the results captured by the top-down camera, and the other columns represent the results captured by four surrounding cameras at 45 degrees. Abnormal images are highlighted for visualization, and normal images are displayed with normal brightness. It can be seen that multi-view shooting can solve the problem of invisibility of abnormal areas from a single viewpoint.
  • Figure 4: Anomalies Visualization of Real-IAD. Each row represents an anomaly class from different products and diverse locations. From top to bottom, they are pit, deformation, abrasion, scratch, damage, missing, foreign objects, and contamination, respectively.
  • Figure 5: Real-IAD Visualisation. (a) The upper six rows show 12 complex objects and all 5 views of each object. (b) The bottom 4 rows show the remaining 18 simple objects and each object shows two views. All anomaly regions are marked in red. Best viewed in zoom and color.