TwinDiffusion: Enhancing Coherence and Efficiency in Panoramic Image Generation with Diffusion Models

TL;DR

TwinDiffusion addresses seam artifacts and efficiency bottlenecks in panoramic image generation with diffusion models by introducing Crop Fusion, a training-free stage that aligns adjacent crop areas, and Cross Sampling, an interleaved, multi-group sampling scheme that permits larger view strides without sacrificing coherence. The method formalizes crop-level fusion via a closed-form, KKT-based optimization that couples denoising guidance with overlap consistency, and leverages a cross-group sampling strategy to accelerate panorama synthesis. Extensive both qualitative and quantitative evaluations show improved panorama coherence (lower LPIPS and DISTS) with competitive fidelity (FID and IS) and faster generation times compared with baselines like MultiDiffusion. The approach demonstrates strong potential for high-quality, efficient panoramic synthesis in applications such as immersive VR and digital art, while acknowledging limitations in global layout stability and social implications of image generation. Overall, TwinDiffusion advances seamless, scalable panoramic diffusion by marrying a lightweight crop fusion mechanism with an efficient interleaved sampling paradigm.

Abstract

Diffusion models have emerged as effective tools for generating diverse and high-quality content. However, their capability in high-resolution image generation, particularly for panoramic images, still faces challenges such as visible seams and incoherent transitions. In this paper, we propose TwinDiffusion, an optimized framework designed to address these challenges through two key innovations: the Crop Fusion for quality enhancement and the Cross Sampling for efficiency optimization. We introduce a training-free optimizing stage to refine the similarity of adjacent image areas, as well as an interleaving sampling strategy to yield dynamic patches during the cropping process. A comprehensive evaluation is conducted to compare TwinDiffusion with the prior works, considering factors including coherence, fidelity, compatibility, and efficiency. The results demonstrate the superior performance of our approach in generating seamless and coherent panoramas, setting a new standard in quality and efficiency for panoramic image generation.

Figures (10)

• Figure 1: TwinDiffusion is a crop-wise framework designed for high-resolution panorama generation with diffusion models. Inspired by the strong connection between twins, our approach aims to reconcile adjacent areas of the panoramic image space successively. This alignment produces pairs of locally similar image crops resembling twins (left), leading to improved coherence and smoother transitions in panoramas (right).
• Figure 2: Illustration of our approach applied to panorama generation. The process begins with the mapping function $F_i$ transforming the image crops into the panoramic space $Z'$. This results in a sequence of overlapping crops $z_t^1,z_t^2,\ldots,z_t^n$ arranged spatially, each having an independent denoising path. Our goal is to optimize $z_t^i$ within the constraints of its adjacent neighbor and itself as well, thus ensuring a unified and progressive fusion of crops. To achieve this alignment, our objective function Eq. \ref{['eq:loss']} is defined into two mutual-restricted parts and reaches the minimizer ${z_t^i}^*$ in each denoising timestep: (i) the matching term: differences at the overlaps of $z_t^i$ and its neighbor $z_t^{i-1}$, (ii) the regularization term: deviations between ${z_t^i}^*$ and its unoptimized self $f_\Phi(F_i(z'_{t+1}),t,C)$.
• Figure 3: Applying our Crop Fusion method to generate twin images. Top: the optimized $I_2^*$ exhibits a seamless fusion effect that meets our expectations. Bottom: We further test its limits by fixing the regularization term of Eq. \ref{['eq:loss_st']}. The results demonstrate our method's robustness under extreme conditions.
• Figure 4: Qualitative comparisons between MultiDiffusion and ours. Our approach significantly reduces the odd joints and visible seams that commonly occur in MultiDiffusion, resulting in higher-quality panoramic images.
• Figure 5: The analysis of the proper timestep $\tau$ for introducing the Crop Fusion stage. As $\tau$ decreases, a gradual transition from under-optimization to over-optimization can be observed, with the best results attained by $\tau=T/2$. The bottom rows of the figure offer two additional examples that further support our findings.