Material Recognition in the Wild with the Materials in Context Database

TL;DR

The paper introduces MINC, a large-scale, open dataset for material recognition in real-world images, built from OpenSurfaces and Houzz with crowdsourced annotations. It demonstrates a CNN-based workflow for patch-level material classification and extends to full-scene, per-pixel segmentation by converting CNNs to fully convolutional networks and integrating a dense CRF. Key findings show strong patch-level accuracy (up to 85.2% mean class) and competitive per-pixel segmentation (73.1% mean class) on real-world imagery, underscoring the importance of large, well-sampled data and context. MINC enables improved material recognition and segmentation and provides a foundation for future work in material attributes and joint material-object understanding.

Abstract

Recognizing materials in real-world images is a challenging task. Real-world materials have rich surface texture, geometry, lighting conditions, and clutter, which combine to make the problem particularly difficult. In this paper, we introduce a new, large-scale, open dataset of materials in the wild, the Materials in Context Database (MINC), and combine this dataset with deep learning to achieve material recognition and segmentation of images in the wild. MINC is an order of magnitude larger than previous material databases, while being more diverse and well-sampled across its 23 categories. Using MINC, we train convolutional neural networks (CNNs) for two tasks: classifying materials from patches, and simultaneous material recognition and segmentation in full images. For patch-based classification on MINC we found that the best performing CNN architectures can achieve 85.2% mean class accuracy. We convert these trained CNN classifiers into an efficient fully convolutional framework combined with a fully connected conditional random field (CRF) to predict the material at every pixel in an image, achieving 73.1% mean class accuracy. Our experiments demonstrate that having a large, well-sampled dataset such as MINC is crucial for real-world material recognition and segmentation.

Figures (10)

• Figure 1: Overview.(a) We construct a new dataset by combining OpenSurfaces bell13opensurfaces with a novel three-stage Amazon Mechanical Turk (AMT) pipeline. (b) We train various CNNs on patches from MINC to predict material labels. (c) We transfer the weights to a fully convolutional CNN to efficiently generate a probability map across the image; we then use a fully connected CRF to predict the material at every pixel.
• Figure 2: Example patches from all 23 categories of the Materials in Context Database (MINC). Note that we sample patches so that the patch center is the material in question (and not necessarily the entire patch). See Table \ref{['tab:patchcount']} for the size of each category.
• Figure 3: AMT pipeline schematic for collecting clicks.(a) Workers filter by images that contain a certain material, (b) workers click on materials, and (c) workers validate click locations by re-labeling each point. Example responses are shown in orange.
• Figure 4: Pipeline for full scene material classification. An image (a) is resized to multiple scales $[1/\sqrt{2}, 1, \sqrt{2}]$. The same sliding CNN predicts a probability map (b) across the image for each scale; the results are upsampled and averaged. A fully connected CRF predicts a final label for each pixel (c). This example shows predictions from a single GoogLeNet converted into a sliding CNN (no average pooling).
• Figure 5: Varying patch scale. We train/test patches of different scales (the patch locations do not vary). The optimum is a trade-off between context and spatial resolution. CNN: AlexNet.