CAD2Render: A Modular Toolkit for GPU-accelerated Photorealistic Synthetic Data Generation for the Manufacturing Industry

TL;DR

CAD2Render tackles data scarcity in manufacturing ML by introducing a modular, GPU-accelerated synthetic-data generator built on Unity HDRP that produces photorealistic, annotated datasets with extensive variations. The system supports camera, object, lighting, and appearance randomization, leveraging path-traced rendering and speedups via DLSS/denoisers while enabling easy export/import in the BOP format for benchmarking and digital twins. Validation on bin picking and 2D keypoint detection demonstrates that models trained solely on synthetic data can achieve industrially relevant accuracy, including near $1\,\mathrm{mm}$ XY pose precision in controlled setups and transferable keypoint localization to real images. Overall, CAD2Render offers a practical, extensible path to scalable, domain-randomized synthetic data generation that helps bridge the sim-to-real gap in manufacturing and beyond.

Abstract

The use of computer vision for product and assembly quality control is becoming ubiquitous in the manufacturing industry. Lately, it is apparent that machine learning based solutions are outperforming classical computer vision algorithms in terms of performance and robustness. However, a main drawback is that they require sufficiently large and labeled training datasets, which are often not available or too tedious and too time consuming to acquire. This is especially true for low-volume and high-variance manufacturing. Fortunately, in this industry, CAD models of the manufactured or assembled products are available. This paper introduces CAD2Render, a GPU-accelerated synthetic data generator based on the Unity High Definition Render Pipeline (HDRP). CAD2Render is designed to add variations in a modular fashion, making it possible for high customizable data generation, tailored to the needs of the industrial use case at hand. Although CAD2Render is specifically designed for manufacturing use cases, it can be used for other domains as well. We validate CAD2Render by demonstrating state of the art performance in two industrial relevant setups. We demonstrate that the data generated by our approach can be used to train object detection and pose estimation models with a high enough accuracy to direct a robot. The code for CAD2Render is available at https://github.com/EDM-Research/CAD2Render.

Figures (10)

• Figure 1: Annotated training set examples of different use cases generated with CAD2Render. left: CNC fabrication, middle: compressor parts, right: tool detection.
• Figure 2: Schematic overview of CAD2Render. Environment maps, CAD files and material properties are imported in CAD2Render and used to create a high variety of 3D scene's. Further variations to the material textures can be created by introducing defects like rust or scratches. The scenes are then rendered with a path tracer to create a dataset for training machine learning algorithms.
• Figure 3: Complex lighting variations. Left: inter-reflections between object and and pallet. Middle: intense highlights (top) and hard shadows (bottom). Right: changes in environment lighting.
• Figure 4: Example of rust and scratch variations. An extracted path material from a physical example (top left) is used in combination with a CAD file (bottom left) to generate synthetic images (middle). On top of the extracted material, we can add different material effects such as rust and scratches (right).
• Figure 5: Texture resampling. Left: starting material texture. Middle: first iterations of copied patches. Some noticeable hard edges can be perceived. Right: refined result after 15 iterations.