RRNet: Configurable Real-Time Video Enhancement with Arbitrary Local Lighting Variations

TL;DR

RRNet introduces a lightweight, real-time video enhancement framework that uses virtual lighting parameter regression combined with a depth-aware rendering module to achieve localized, identity-preserving relighting under uneven illumination. By predicting a minimal set of virtual light sources and applying a Blinn–Phong–style renderer, RRNet delivers high-quality results with significantly reduced computational cost compared to pixel-wise, encoder–decoder approaches. The model is trained with a generative AI-based FFHQL dataset that provides unpaired, diverse lighting scenarios, enabling scalable training and robust generalization. Experimental results show RRNet outperforms prior methods on low-light enhancement, localized illumination adjustment, and glare removal, while maintaining temporal coherence and real-time performance on 1080p video, making it suitable for video conferencing, AR portrait enhancement, and mobile photography.

Abstract

With the growing demand for real-time video enhancement in live applications, existing methods often struggle to balance speed and effective exposure control, particularly under uneven lighting. We introduce RRNet (Rendering Relighting Network), a lightweight and configurable framework that achieves a state-of-the-art tradeoff between visual quality and efficiency. By estimating parameters for a minimal set of virtual light sources, RRNet enables localized relighting through a depth-aware rendering module without requiring pixel-aligned training data. This object-aware formulation preserves facial identity and supports real-time, high-resolution performance using a streamlined encoder and lightweight prediction head. To facilitate training, we propose a generative AI-based dataset creation pipeline that synthesizes diverse lighting conditions at low cost. With its interpretable lighting control and efficient architecture, RRNet is well suited for practical applications such as video conferencing, AR-based portrait enhancement, and mobile photography. Experiments show that RRNet consistently outperforms prior methods in low-light enhancement, localized illumination adjustment, and glare removal.

Figures (6)

• Figure 1: Result on the VCD dataset running on an NVIDIA GeForce RTX 3090 GPU. Lower NIQE scores indicate better quality, while shorter processing times suggest higher efficiency. For reference, the runtime of RRNet (1-frame), RRNet (3-frame), and RRNet (10-frame) are 17.0 ms, 6.3 ms, and 2.6 ms per frame, respectively.
• Figure 2: Visual comparisons of various low-light enhancement methods on test images from the FFHQ, VV, and VCD datasets show that RRNet achieves superior exposure balance and preserves skin tones more effectively than other state-of-the-art methods.
• Figure 3: Architecture of RRNet. (a) Overall framework with Lighting Parameter Regression Module (LPRM), Rendering Module (RM), and optional Albedo Generation Module (AGM); the dashed line in LPRM indicates shared weights between encoders $E_0$ and $E_1$. (b) RM details; $M$, $N$, and $\theta^*$ denote depth, surface normal, and estimated lighting parameters.
• Figure 4: Examples of different lighting conditions generated for the same portrait in the FFHQL dataset, including variations in light direction, pattern, intensity, and color temperature. Boxed are the GTs selected by vote.
• Figure 5: Comparison of frame enhancement on an unbalanced exposure video from the VCD dataset. From top to bottom: input frames and results of ZeroDCE++, SNRNet, StableLLVE, and our method.