MVIP -- A Dataset and Methods for Application Oriented Multi-View and Multi-Modal Industrial Part Recognition

TL;DR

The paper presents MVIP, a comprehensive, calibrated multi-view and multi-modal dataset for industrial part recognition that combines RGBD views with object properties, natural language tags, and super-classes. It introduces a modular MV/MM framework with various fusion strategies and a novel multi-head auxiliary loss to encourage balanced contributions from views and modalities, demonstrating strong baseline performance and providing insights into fusion design and data augmentation. The dataset, captured with ten RGBD cameras and a scale to enable 3D reconstruction, supports synthetic data generation and incremental research toward deployment-ready industrial classifiers. Key findings include the stability and accuracy gains from MH-loss, the potential of Transformer-based MV fusion with proper pretraining, and the challenges of RGBD fusion due to pretraining mismatches, informing future work in multi-modal pre-training and industrial data curation.

Abstract

We present MVIP, a novel dataset for multi-modal and multi-view application-oriented industrial part recognition. Here we are the first to combine a calibrated RGBD multi-view dataset with additional object context such as physical properties, natural language, and super-classes. The current portfolio of available datasets offers a wide range of representations to design and benchmark related methods. In contrast to existing classification challenges, industrial recognition applications offer controlled multi-modal environments but at the same time have different problems than traditional 2D/3D classification challenges. Frequently, industrial applications must deal with a small amount or increased number of training data, visually similar parts, and varying object sizes, while requiring a robust near 100% top 5 accuracy under cost and time constraints. Current methods tackle such challenges individually, but direct adoption of these methods within industrial applications is complex and requires further research. Our main goal with MVIP is to study and push transferability of various state-of-the-art methods within related downstream tasks towards an efficient deployment of industrial classifiers. Additionally, we intend to push with MVIP research regarding several modality fusion topics, (automated) synthetic data generation, and complex data sampling -- combined in a single application-oriented benchmark.

Figures (6)

• Figure 1: An indexed ROI cropped MVIP image set featuring ten simultaneously captured Views
• Figure 2: Digitisation station used for MVIP
• Figure 3: Simple 3D-Object-Reconstruction given a single image set from MVIP.
• Figure 4: Subset of industrial parts featured in the MVIP dataset.
• Figure 5: Multi-view architecture used to train and evaluate our experiments, please find further description within our methods.