StochSync: Stochastic Diffusion Synchronization for Image Generation in Arbitrary Spaces

TL;DR

StochSync addresses zero-shot generation in arbitrary canonical spaces by unifying Diffusion Synchronization (DS) and Score Distillation Sampling (SDS). It introduces maximum stochasticity, multi-step $x_{0|t}$ estimation via a denoiser $\mathcal{G}(\cdot)$, and non-overlapping view sampling to achieve coherent outputs across multiple views without strong conditioning. The method delivers state-of-the-art results for depth-free $360^\circ$ panorama generation and competitive mesh texturing, outperforming finetuning-based and several zero-shot baselines while reducing seam artifacts. By reframing DS and SDS in a unified stochastic optimization perspective, StochSync advances practical generation in spheres, cylinders, tori, and mesh surfaces, with implications for wide-ranging AR/VR and graphics tasks.

Abstract

We propose a zero-shot method for generating images in arbitrary spaces (e.g., a sphere for 360° panoramas and a mesh surface for texture) using a pretrained image diffusion model. The zero-shot generation of various visual content using a pretrained image diffusion model has been explored mainly in two directions. First, Diffusion Synchronization-performing reverse diffusion processes jointly across different projected spaces while synchronizing them in the target space-generates high-quality outputs when enough conditioning is provided, but it struggles in its absence. Second, Score Distillation Sampling-gradually updating the target space data through gradient descent-results in better coherence but often lacks detail. In this paper, we reveal for the first time the interconnection between these two methods while highlighting their differences. To this end, we propose StochSync, a novel approach that combines the strengths of both, enabling effective performance with weak conditioning. Our experiments demonstrate that StochSync provides the best performance in 360° panorama generation (where image conditioning is not given), outperforming previous finetuning-based methods, and also delivers comparable results in 3D mesh texturing (where depth conditioning is provided) with previous methods.

Figures (37)

• Figure 1: Assorted mesh textures and panoramas generated using StochSync, including one in the background (environment map), which is a 360° panorama. StochSync extends the capabilities of image diffusion models trained in square spaces to produce images in arbitrary spaces such as cylinders, spheres, tori, and mesh surfaces.
• Figure 2: A comparison of SyncTweedies Kim2024:SyncTweedies, a synchronization method, SDS Poole:2023DreamFusion, and StochSync which uses SyncTweedies as a base and incorporates maximum stochasticity (Max $\sigma_t$), multi-step $\mathbf{x}_{0|t}$ computation (Impr. $\mathbf{x}_{0|t}$), and non-overlapping view sampling (N.O. Views), alongside others that use only a subset of these components.
• Figure 3: Qualitative results of panorama generation using PanFusion Zhang2024:PanFusion prompts. Comparisons to previous works are presented in the left column, while the ablation cases are shown in the right column along with StochSync.
• Figure 4: Qualitative result of 3D mesh texturing. StochSync generates realistic texture images, demonstrating its applicability even in the conditional generation case.
• Figure 5: 3D mesh textures on spheres and tori generated by StochSync.

Theorems & Definitions (1)

• proof