ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Research

TL;DR

ICONIC-444 addresses a key bottleneck in OOD detection by providing a large, contamination-free industrial dataset with 444 ID classes and over $3.1$ million images, enabling controlled evaluation across near-, far-, extreme-, and synthetic-OOD. The authors define four ID tasks and benchmark 22 post-hoc methods using two backbones, revealing that feature-based detectors often outperform baselines and that even state-of-the-art approaches struggle at $TPR$ of $99\%$, especially for near- and far-OOD. The result is a nuanced view: dataset characteristics strongly influence which OOD methods perform best, with GRAM, ViM, and KNN excelling in this industrial, fine-grained setting, while CLIP-based approaches underperform. ICONIC-444 thus offers a valuable, domain-specific platform for robust, statistically meaningful OOD evaluation and fosters progress toward reliable deployment in safety- and sustainability-critical industrial applications.

Abstract

Current progress in out-of-distribution (OOD) detection is limited by the lack of large, high-quality datasets with clearly defined OOD categories across varying difficulty levels (near- to far-OOD) that support both fine- and coarse-grained computer vision tasks. To address this limitation, we introduce ICONIC-444 (Image Classification and OOD Detection with Numerous Intricate Complexities), a specialized large-scale industrial image dataset containing over 3.1 million RGB images spanning 444 classes tailored for OOD detection research. Captured with a prototype industrial sorting machine, ICONIC-444 closely mimics real-world tasks. It complements existing datasets by offering structured, diverse data suited for rigorous OOD evaluation across a spectrum of task complexities. We define four reference tasks within ICONIC-444 to benchmark and advance OOD detection research and provide baseline results for 22 state-of-the-art post-hoc OOD detection methods.

Figures (17)

• Figure 1: Overview of ICONIC-444 and its application in creating tasks for image classification and OOD detection evaluation. Top panel: An image grid illustrating the progression from fine-grained to coarse-grained ID classes alongside OOD samples of varying difficulty levels, near, far, extreme (from external datasets such as ImageNet deng2009_imagenet, iNaturalist Horn_2018_inat, Places365 zhou2017places, and Textures cimpoi14dtd), as well as synthetic images, providing diverse OOD detection benchmarks. Bottom panel: A histogram on a logarithmic scale showing the distribution of images per class. Classes are organized left to right by category (Food, Non-Food), group size (total images per group), and then by the number of images per class within each group.
• Figure 2: Visualization of data samples (bottom row) from the Wheat task, showing the 12 ID classes and all OOD categories (near, far, extreme, and synthetic) and their embedded representations (top) extracted from the ResNet18 penultimate layer. The t-SNE visualizations (top) include only a limited subset of samples to enhance readability. The bottom row shows representative images from each category, highlighting the sample diversity. For ID data, one sample per class is shown; for OOD data, only a small subset of classes is depicted.
• Figure 3: Comparison of the mean FPR95 and FPR99 for each OOD detection method, averaged over three random seeds, all four tasks, and all four OOD categories. Darker shades present the FPR at $95\%$ TPR, while the lighter shades show the gap to the more challenging (but highly relevant in practice) FPR at $99\%$ TPR.
• Figure 4: Difficult near- and far-OOD samples for GRAM sastry20a_gram on the Almond task (ResNet18). The first row shows example images from the OOD classes that are most frequently misclassified as ID, while the second row presents the closest (in feature space) ID (almond) classes causing these confusions.
• Figure 5: Views of the sorting machine prototype. (a) External overview, highlighting the vibrating conveyor, blue LED backlight, white LED lights, and camera housing. (b) Internal perspective, illustrating the blue LED backlight, white LED lights, chute, and camera within the machine's main structure.