Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

mitransient: Transient light transport in Mitsuba 3

Diego Royo, Jorge Garcia-Pueyo, Miguel Crespo, Óscar Pueyo-Ciutad, Guillermo Enguita, Diego Bielsa

TL;DR

mitransient introduces a time-resolved extension to Mitsuba 3, providing a practical, Python-based platform for simulating transient light transport with CPU/GPU support and easy installation via PyPi. It couples two transient path tracing integrators with time-domain polarization tracking using a Stokes vector and Mueller matrices, and adds differentiable rendering in the temporal domain for both forward and inverse problems. A dedicated NLOS pipeline with transient_nlos_path and an accompanying toolkit (y-tal) enables rapid, realistic reconstructions and hardware-noise modeling, with substantial speedups demonstrated on representative scenes. Collectively, the work lowers barriers to developing and evaluating transient imaging algorithms, enabling rapid prototyping, optimization, and hardware-agnostic testing across both standard and non-line-of-sight scenarios.

Abstract

mitransient is a light transport simulation tool that extends Mitsuba 3 with support for time-resolved simulations. In essence, mitransient extends conventional rendering by adding a temporal dimension which accounts for the time of flight of light. This allows rapid prototyping of novel transient imaging systems without the need of costly or difficult-to-operate hardware. Our code is trivially easy to install through pip, and consists of Python modules that can run both in CPU and GPU by leveraging the JIT capabilities of Mitsuba 3. It provides physically-based simulations of complex phenomena, including a wide variety of realistic materials and participating media such as fog or smoke. In addition, we extend Mitsuba 3's functionality to support time-resolved polarization tracking of light and transient differentiable rendering. Finally, we also include tools that simplify the use of our simulations for non-line-of-sight imaging, enabling realistic scene setups with capture noise to be simulated in just seconds of minutes. Altogether, we hope that mitransient will support the research community in developing novel algorithms for transient imaging.

mitransient: Transient light transport in Mitsuba 3

TL;DR

mitransient introduces a time-resolved extension to Mitsuba 3, providing a practical, Python-based platform for simulating transient light transport with CPU/GPU support and easy installation via PyPi. It couples two transient path tracing integrators with time-domain polarization tracking using a Stokes vector and Mueller matrices, and adds differentiable rendering in the temporal domain for both forward and inverse problems. A dedicated NLOS pipeline with transient_nlos_path and an accompanying toolkit (y-tal) enables rapid, realistic reconstructions and hardware-noise modeling, with substantial speedups demonstrated on representative scenes. Collectively, the work lowers barriers to developing and evaluating transient imaging algorithms, enabling rapid prototyping, optimization, and hardware-agnostic testing across both standard and non-line-of-sight scenarios.

Abstract

mitransient is a light transport simulation tool that extends Mitsuba 3 with support for time-resolved simulations. In essence, mitransient extends conventional rendering by adding a temporal dimension which accounts for the time of flight of light. This allows rapid prototyping of novel transient imaging systems without the need of costly or difficult-to-operate hardware. Our code is trivially easy to install through pip, and consists of Python modules that can run both in CPU and GPU by leveraging the JIT capabilities of Mitsuba 3. It provides physically-based simulations of complex phenomena, including a wide variety of realistic materials and participating media such as fog or smoke. In addition, we extend Mitsuba 3's functionality to support time-resolved polarization tracking of light and transient differentiable rendering. Finally, we also include tools that simplify the use of our simulations for non-line-of-sight imaging, enabling realistic scene setups with capture noise to be simulated in just seconds of minutes. Altogether, we hope that mitransient will support the research community in developing novel algorithms for transient imaging.

Paper Structure

This paper contains 7 sections, 4 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Transient rendering
  3. Transient polarization tracking
  4. Differentiable transient rendering
  5. Non-line-of-sight (NLOS) scene capture simulation
  6. y-tal: A software toolkit for NLOS captures.
  7. Realistic hardware noise.

Figures (2)

  • Figure 1: Rendering polarization in transient rendering. We support in mitransient rendering the polarization state of transient light transport, by capturing the Stokes vector of radiance reaching the sensor. In the Spaceship scene, the polarization state at different frames varies due to the propagation delays between interactions of light in the spaceship. The angle of linear uses a rainbow colormap and the degree polarization a scale of reds, following the convention on polarized rendering WilkiePolvis10.
  • Figure 2: In order to showcase our system in a NLOS scenario, we have recreated the Office scene from Liu et al. Liu2019phasor. (a) The laser emits a pulse towards the relay wall, and the ultra-fast camera captures the response of the hidden scene. We show one pixel of this time-resolved response. (b) NLOS reconstructions of the captured and synthetic scenes.