VistaFlow: Photorealistic Volumetric Reconstruction with Dynamic Resolution Management via Q-Learning

TL;DR

VistaFlow addresses the trade-off between visual fidelity and rendering speed in 3D reconstruction from 2D photos. It introduces a PlenOctree-based differentiable renderer and a Q-learning driven dynamic resolution controller, QuiQ, to adapt sampling density in real time. By bypassing NeRFs and using PlenOctree, VistaFlow achieves real-time rendering on CPU-based platforms while maintaining photorealistic quality, with reported 1080p at over 100 FPS on consumer hardware. The work demonstrates substantial FPS gains with competitive perceptual metrics, enabling scalable, accessible volumetric rendering across a range of devices.

Abstract

We introduce VistaFlow, a scalable three-dimensional imaging technique capable of reconstructing fully interactive 3D volumetric images from a set of 2D photographs. Our model synthesizes novel viewpoints through a differentiable rendering system capable of dynamic resolution management on photorealistic 3D scenes. We achieve this through the introduction of QuiQ, a novel intermediate video controller trained through Q-learning to maintain a consistently high framerate by adjusting render resolution with millisecond precision. Notably, VistaFlow runs natively on integrated CPU graphics, making it viable for mobile and entry-level devices while still delivering high-performance rendering. VistaFlow bypasses Neural Radiance Fields (NeRFs), using the PlenOctree data structure to render complex light interactions such as reflection and subsurface scattering with minimal hardware requirements. Our model is capable of outperforming state-of-the-art methods with novel view synthesis at a resolution of 1080p at over 100 frames per second on consumer hardware. By tailoring render quality to the capabilities of each device, VistaFlow has the potential to improve the efficiency and accessibility of photorealistic 3D scene rendering across a wide spectrum of hardware, from high-end workstations to inexpensive microcontrollers.

Figures (5)

• Figure 1: Overview of the QuiQ Training Process. We begin with a trained PlenOctree model as described in Fig 1. We then (a) benchmark the system by rendering the model along a predetermined set of camera movements. During this process, we collect framerate information to assess the effect of various rendering parameters on output quality. Next, we (b) use a k-Nearest Neighbors algorithm to find the most similar data from a (c) selection of prerecorded benchmark profiles. This allows us to effectively increase our dataset without the necessary collection time. We use this increased dataset to (d) train a reward function through ridge regression. Finally, this reward function is used to train the (e) dynamic resolution controller that ultimately controls the resolution parameters during output.
• Figure 2: Results from our direct PlenOctree training method. We demonstrate efficient model optimization in minutes. Shown above is “drums” from the NeRF Synthetic dataset.
• Figure 3: Quantitative results on NeRF Synthetic scenes. VistaFlow outperforms every previous method in both PSNR (which represents model accuracy) and FPS
• Figure 4: Training curves on the NeRF Synthetic drums scene. We find that VistaFlow trains both faster and more efficiently than previous methods. We maintain this significant gap in PSNR even as training time increases.
• Figure 5: QuiQ Activations per second at varying levels of computational load as measured by CPU usage. QuiQ adjusts ray sampling parameters so rapidly that changes are invisible in the final output. This frequent activation is what allows QuiQ to achieve such impressive PSNR, SSIM, and LPIPS scores.