SyncTweedies: A General Generative Framework Based on Synchronized Diffusions

TL;DR

A previously unexplored case is revealed: averaging the outputs of Tweedie's formula while conducting denoising in multiple instance spaces, which provides the best quality with the widest applicability to downstream tasks.

Abstract

We introduce a general framework for generating diverse visual content, including ambiguous images, panorama images, mesh textures, and Gaussian splat textures, by synchronizing multiple diffusion processes. We present exhaustive investigation into all possible scenarios for synchronizing multiple diffusion processes through a canonical space and analyze their characteristics across applications. In doing so, we reveal a previously unexplored case: averaging the outputs of Tweedie's formula while conducting denoising in multiple instance spaces. This case also provides the best quality with the widest applicability to downstream tasks. We name this case SyncTweedies. In our experiments generating visual content aforementioned, we demonstrate the superior quality of generation by SyncTweedies compared to other synchronization methods, optimization-based and iterative-update-based methods.

Figures (16)

• Figure 1: Diverse visual content generated by SyncTweedies: A diffusion synchronization process applicable to various downstream tasks without finetuning.
• Figure 2: Diagrams of diffusion synchronization processes. The left diagram depicts denoising instance variables $\{\mathbf{w}_i\}$, while the right diagram illustrates directly denoising a canonical variable $\mathbf{z}$.
• Figure 3: Qualitative results of ambiguous image generation. While all diffusion synchronization processes show identical results with $1$-to-$1$ projections, Case 1, Case 3 and Visual Anagrams Geng:2023VisualAnagrams (Case 4) exhibit degraded performance when the projections are $1$-to-$n$. Notably, SyncTweedies can be applied to the widest range of projections, including $n$-to-$1$ projections, where Case 5 fails to generate plausible outputs.
• Figure 4: Qualitative results of 3D mesh texturing.SyncTweedies and SyncMVD Liu2023:SyncMVD generate realistic texture images, achieving better results than other baselines including finetuning-based method. Other diffusion synchronization cases fail to produce plausible textures.
• Figure 5: Qualitative results of 3D Gaussian splats Kerbl2023:3DGS texturing.[S$^*$] is a prefix prompt. We use "Make it to" for IN2N Haque2023:IN2N and "A photo of" for the others. SyncTweedies generates high-fidelity textures, while Case 5 lacks fine details due to the variance reduction issue.