What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?

TL;DR

The paper investigates why LoRA-based fine-tuning of pre-trained diffusion models can generate high-quality 360-degree panoramas. By decomposing cross-attention into $Q=z_t W_q$, $K=y W_k$, and $V=y W_v$ with $\mathrm{MHCA}(z_t,y)$, it shows that $W_q$/$W_k$ learn shared, non-panoramic information while $W_v$/$W_o$ specialize for equirectangular panorama structure. Based on these insights, it proposes UniPano — a memory-efficient uni-branch approach that freezes $W_q$/$W_k$ and trains $W_v$/$W_o$ (often with mixture-of-experts) — achieving state-of-the-art FAED and strong FID with lower training cost than dual-branch methods. The method scales to higher resolutions using Stable Diffusion 3, enabling end-to-end panorama generation with improved efficiency and robustness for complex prompts, albeit with some failure cases such as invalid layouts. Overall, the work provides both a mechanistic understanding of panorama adaptation and a practical baseline that accelerates high-resolution text-to-panorama generation.

Abstract

Recent prosperity of text-to-image diffusion models, e.g. Stable Diffusion, has stimulated research to adapt them to 360-degree panorama generation. Prior work has demonstrated the feasibility of using conventional low-rank adaptation techniques on pre-trained diffusion models to generate panoramic images. However, the substantial domain gap between perspective and panoramic images raises questions about the underlying mechanisms enabling this empirical success. We hypothesize and examine that the trainable counterparts exhibit distinct behaviors when fine-tuned on panoramic data, and such an adaptation conceals some intrinsic mechanism to leverage the prior knowledge within the pre-trained diffusion models. Our analysis reveals the following: 1) the query and key matrices in the attention modules are responsible for common information that can be shared between the panoramic and perspective domains, thus are less relevant to panorama generation; and 2) the value and output weight matrices specialize in adapting pre-trained knowledge to the panoramic domain, playing a more critical role during fine-tuning for panorama generation. We empirically verify these insights by introducing a simple framework called UniPano, with the objective of establishing an elegant baseline for future research. UniPano not only outperforms existing methods but also significantly reduces memory usage and training time compared to prior dual-branch approaches, making it scalable for end-to-end panorama generation with higher resolution. The code will be released.

Figures (13)

• Figure 1: Our UniPano can synthesize realistic 360-degree panoramic images by fine-tuning Stable Diffusion. (Top) $1024\times 2048$ panoramic images generated by UniPano. (Bottom) $512\times 1024$ panoramic images generated by UniPano.
• Figure 2: Illustration of fine-tuning cross-attention blocks within diffusion models with low-rank adaptation (LoRA). MCA, MatMul, $h$ denotes the multi-head cross attention, matrix multiplication, and the number of attention heads respectively.
• Figure 3: Qualitative comparison for training $W_{\{q,k,v,o\}}$ in isolation separately. Training $W_q$ or $W_k$ in isolation fails to capture the spherical structure, as in (a) and (b); whereas training $W_v$ or $W_o$ in isolation successfully captures the spherical distortion of the panoramic images, as in (c) and (d). All visualizations are generated with the text prompt "a kitchen with stainless steel appliances".
• Figure 4: Illustration of roles of $W_{\{q,k,v,o\}}$ when jointly fine-tuned. We first fine-tune $W_{\{q,k,v,o\}}$ jointly with LoRA on panoramic data, then optionally deactivate some LoRAs for inference with different purposes. (a) the panoramic image generated by fine-tuning $W_{\{q,k,v,o\}}$ jointly with LoRA, where we simplify the LoRA architecture (cf.\ref{['fig:attn-lora']}); (b) diffusion models with panorama fine-tuned LoRAs can only generate distorted, panoramic-like images, are thus no longer capable of generating perspective images; (c) by excluding $W_v$ and $W_o$ LoRAs, diffusion models fine-tuned on panoramic data recover the ability to generate perspective images; (d) excluding $W_q$ and $W_k$ LoRAs does not affect the model's ability to generate panoramic images.
• Figure 5: Selected qualitative comparisons between UniPano (Ours) and PanFusion. We show the generated panoramic image for each text prompt and 4 randomly sampled horizontal perspective views below. We highlight notable artifacts such as non-perspective lines with red boxes. More qualitative comparisons can be found in the supplementary material.

Theorems & Definitions (2)

• Remark 3.3
• Remark 3.6