SyncLight: Controllable and Consistent Multi-View Relighting

TL;DR

SyncLight addresses the challenge of consistent, parametric relighting across multiple uncalibrated views of a static scene by introducing a latent-bridge diffusion framework with a multi-view transformer. It propagates lighting edits from a single reference view in one forward pass, without requiring camera poses, enabling zero-shot generalization to arbitrary view counts. The authors also contribute the SyncLight dataset, blending synthetic and real multi-view captures with controlled illumination, to train and benchmark cross-view relighting. The approach yields strong cross-view consistency, efficient inference, and practical workflows for multi-camera broadcasts, virtual production, and video relighting, while enabling integration with radiance-field methods for 3D-aware relighting.

Abstract

We present SyncLight, the first method to enable consistent, parametric relighting across multiple uncalibrated views of a static scene. While single-view relighting has advanced significantly, existing generative approaches struggle to maintain the rigorous lighting consistency essential for multi-camera broadcasts, stereoscopic cinema, and virtual production. SyncLight addresses this by enabling precise control over light intensity and color across a multi-view capture of a scene, conditioned on a single reference edit. Our method leverages a multi-view diffusion transformer trained using a latent bridge matching formulation, achieving high-fidelity relighting of the entire image set in a single inference step. To facilitate training, we introduce a large-scale hybrid dataset comprising diverse synthetic environments -- curated from existing sources and newly designed scenes -- alongside high-fidelity, real-world multi-view captures under calibrated illumination. Surprisingly, though trained only on image pairs, SyncLight generalizes zero-shot to an arbitrary number of viewpoints, effectively propagating lighting changes across all views, without requiring camera pose information. SyncLight enables practical relighting workflows for multi-view capture systems.

Figures (8)

• Figure 1: SyncLight formulates multi-view relighting as a conditional flow matching problem in latent space. (Left) Input scenes under source ($\mathbf{x}_\text{src}$) and target lighting ground truth ($\mathbf{x}_\text{tar}$) are encoded into latents ($\mathbf{z}_\text{src}, \mathbf{z}_\text{tar}$) via a VAE encoder. This done for both the "Reference" (0) and "Other" (1) views. (Middle) During training, we sample a timestep $t$ and construct a Bridge Matching (BM) interpolant $\mathbf{z}_t$. Our backbone, "Multi-View SD", is conditioned on a user-specified "Lightmap" (encoding color as intensity and chromaticity) derived from the "Reference view". To enforce consistency, the backbone processes both views simultaneously. (Right) The model predicts the target latents, which are decoded into relit images ($\hat{\mathbf{x}}_T$). The network is optimized using a hybrid objective $\mathcal{L}$ combining latent flow matching loss ($\mathcal{L}_{\text{lbm}}$) with pixel-level reconstruction losses ($\mathcal{L}_{\text{pix}}$) for each view to ensure high-fidelity, consistent relighting.
• Figure 2: Multi-view transformer block. To ensure consistent relighting across viewpoints, we modify the standard self-attention mechanism of SDXL. View features are concatenated and reshaped along the sequence dimension, creating a unified representation of shape $[B, 2 T, F]$. This enables the transformer block to perform global self-attention across all views simultaneously, effectively propagating lighting cues from the reference view to target views. This formulation is agnostic to the sequence length: while trained on pairs ($N=2$, as shown here), it supports an arbitrary number of views at inference, enabling zero-shot generalization to $N>2$.
• Figure 3: Parametric color control, with the lamp relit to different colors. In the original captures (not shown), the lamp is turned off. SyncLight correctly relights both the reference and other views consistently across all colors. Note that the color cast on the wall is also consistent with the light color.
• Figure 4: Parametric intensity control. (left to right) The target lamp intensity is set to increasing levels; SyncLight correctly relits both views consistently. Note how the shadow of the flowers progressively increases in the second view (bottom row), even if the conditioning is provided for the reference view (top row).
• Figure 5: Relighting a scene with seven different views. Top row: Input images. Bottom row: SyncLight results. Note how all the views are consistently modified. We want to emphasize views 2 and 4, in which the lamp turned on is not visible, yet the effects in the scene match those of the other views.