REFRAME: Reflective Surface Real-Time Rendering for Mobile Devices

TL;DR

REFRAME tackles real-time novel view synthesis for highly reflective surfaces on mobile devices by distilling a NeRF-derived mesh into a fast, mesh-based renderer augmented with a neural environment map. The approach introduces a geometry learner to robustly refine vertex positions and normals, and a diffuse–specular color formulation that uses a reflection-direction cue together with a learned environment feature map to capture complex reflections without heavy on-device computation. By baking the environment features into a 2D map and textures, REFRAME achieves real-time rendering (>200 FPS on high-end GPUs and mobile-ready performance) while maintaining reconstruction quality on challenging reflective scenes, and enabling relighting and simple scene editing. The work demonstrates competitive to state-of-the-art quality among real-time methods, with notable advantages in foreground reflective fidelity and low memory overhead, albeit with some limitations on background handling and interreflections dependent on the initial mesh quality.

Abstract

This work tackles the challenging task of achieving real-time novel view synthesis for reflective surfaces across various scenes. Existing real-time rendering methods, especially those based on meshes, often have subpar performance in modeling surfaces with rich view-dependent appearances. Our key idea lies in leveraging meshes for rendering acceleration while incorporating a novel approach to parameterize view-dependent information. We decompose the color into diffuse and specular, and model the specular color in the reflected direction based on a neural environment map. Our experiments demonstrate that our method achieves comparable reconstruction quality for highly reflective surfaces compared to state-of-the-art offline methods, while also efficiently enabling real-time rendering on edge devices such as smartphones.

Figures (14)

• Figure 1: Pipeline for REFRAME. Components with a yellow background are either baked or omitted during the mobile rendering stage. Training Stage: The initial mesh is updated first before performing differentiable rasterization difras. Next, we obtain the diffuse color $\mathbf{c}_d$ and specular feature $\mathbf{f}_s$ based on the position $\mathbf{p}$, and the environment feature $\mathbf{f}_e$ based on reflective direction $\boldsymbol{\omega}_{r}$. Then we obtain the specular color $\mathbf{c}_s$ and combine it with $\mathbf{c}_d$ to create the final full color $\mathbf{c}$. Mobile Rendering Stage: We bake the intermediate output for real-time rendering. This mesh-based rendering can be implemented using WebGL and easily deployed on various platforms (e.g., desktop and mobile devices). Here, we retrieve the $\mathbf{c}_d$ and $\mathbf{f}_s$ from baked texture images, and the $\mathbf{f}_e$ from the environment feature map. $\mathbf{c}_d$ and $\mathbf{c}_s$ are processed the same as the training stage to obtain $\mathbf{c}$.
• Figure 2: Mesh Optimization. The geometry of the initial mesh is often poor. We are able to significantly improve the geometry of the mesh through geometry learner.
• Figure 3: Relighting. We can achieve relighting of objects by editing our environment feature map. The image illustrates the editing process of flipping or replacing the environmental feature map. The left side shows the rendering result, the top right corner presents the corresponding environment feature map, and the specular color is displayed in the bottom right corner.
• Figure 4: Rendering Quality on Shiny Blender Dataset. Our method achieves optimal rendering quality in most scenes and provides better modeling of reflective appearance compared to the comparison methods.
• Figure 5: Qualitative Comparison. Compared to the baselines, our method reconstructs reflective regions of higher fidelity and yields more accurate surface geometry.