MatPredict: a dataset and benchmark for learning material properties of diverse indoor objects

TL;DR

MatPredict introduces a dataset and benchmark to learn material properties of indoor objects from images by marrying Replica’s high-fidelity indoor meshes with MatSynth’s extensive material classes, yielding 18 objects across 14 materials with 512 renders per object–material pair under varied lighting and viewpoints. A four-model encoder–decoder benchmark evaluates the recovery of basecolor and roughness from 224×224 crops, demonstrating the feasibility of inverse rendering for indoor robotics using physically based rendering pipelines and UDIM texture mapping. The work provides a scalable, diverse dataset and a practical baseline for material-property inference, with clear extensibility to additional material channels and recognition tasks, while acknowledging domain gaps and current material-channel limitations. This dataset and benchmark can enhance robust material-aware perception and manipulation in consumer robotics by enabling learning under diverse illumination and material configurations.

Abstract

Determining material properties from camera images can expand the ability to identify complex objects in indoor environments, which is valuable for consumer robotics applications. To support this, we introduce MatPredict, a dataset that combines the high-quality synthetic objects from Replica dataset with MatSynth dataset's material properties classes - to create objects with diverse material properties. We select 3D meshes of specific foreground objects and render them with different material properties. In total, we generate \textbf{18} commonly occurring objects with \textbf{14} different materials. We showcase how we provide variability in terms of lighting and camera placement for these objects. Next, we provide a benchmark for inferring material properties from visual images using these perturbed models in the scene, discussing the specific neural network models involved and their performance based on different image comparison metrics. By accurately simulating light interactions with different materials, we can enhance realism, which is crucial for training models effectively through large-scale simulations. This research aims to revolutionize perception in consumer robotics. The dataset is provided \href{https://huggingface.co/datasets/UMTRI/MatPredict}{here} and the code is provided \href{https://github.com/arpan-kusari/MatPredict}{here}.

Figures (6)

• Figure 1: Example glass decorative pieces
• Figure 2: Table rendered by different materials
• Figure 3: Effect of the UV scale factor $s$ on appearance. Smaller $s$ values map a larger texture footprint to the surface, producing finer weave patterns, whereas larger $s$ values stretch the same fabric map, leading to visibly coarser detail.
• Figure 4: Effect of viewpoint on appearance. Five leather-pillow renderings sampled from the latitude–longitude grid demonstrate how changes in elevation $\phi$ and azimuth $\theta$ influence specular highlights, perceived shape, and shadow placement.
• Figure 5: leather pillow with different lighting setup