NViSII: A Scriptable Tool for Photorealistic Image Generation

TL;DR

The paper introduces NViSII, an open-source Python-enabled ray tracer built on NVIDIA OptiX that enables scriptable, photorealistic image generation for synthetic data in computer vision and deep learning. It presents an ECS-based architecture, a Python API, multi-GPU path tracing via OWL, and metadata outputs, all designed for easy domain randomization and high-quality renders. The authors demonstrate utility by training a DOPE pose estimator and a 2D object detector on synthetic data and show improved sim-to-real transfer, including improvements when materials are physically modeled. Compared with existing raster- and game-engine based tools, NVISII emphasizes scriptability, accessible installation via pip, and physics-based lighting to bridge the reality gap.

Abstract

We present a Python-based renderer built on NVIDIA's OptiX ray tracing engine and the OptiX AI denoiser, designed to generate high-quality synthetic images for research in computer vision and deep learning. Our tool enables the description and manipulation of complex dynamic 3D scenes containing object meshes, materials, textures, lighting, volumetric data (e.g., smoke), and backgrounds. Metadata, such as 2D/3D bounding boxes, segmentation masks, depth maps, normal maps, material properties, and optical flow vectors, can also be generated. In this work, we discuss design goals, architecture, and performance. We demonstrate the use of data generated by path tracing for training an object detector and pose estimator, showing improved performance in sim-to-real transfer in situations that are difficult for traditional raster-based renderers. We offer this tool as an easy-to-use, performant, high-quality renderer for advancing research in synthetic data generation and deep learning.

Figures (10)

• Figure 1: Our scriptable tool leverages hardware-accelerated path tracing to generate photorealistic images. Annotations are shown along the bottom row, from left to right: image with no motion blur, depth, surface normals, segmentation labels, texture coordinates, and optical flow.
• Figure 2: Example renders. Top: bistro scene$^{1}$ (zoom for details), indoor scene$^{2}$, texture used as light. Bottom: smoke with 3D volume, reflective metal, domain randomization.
• Figure 3: A minimal Python script example.
• Figure 4: Different material configurations. From left to right: material with roughness = 1.0 (velvet), roughness = 0.0 (plastic), transmission = 1.0 (glass), metallic = 1.0 (metal).
• Figure 5: Motion blur and defocus blur.