Untwisting RoPE: Frequency Control for Shared Attention in DiTs

TL;DR

RoPE decomposes attention in diffusion transformers into frequency components, with high-frequency channels enforcing locality and driving reference copying in shared-attention setups. The authors introduce a frequency-aware RoPE modulation that attenuates high-frequency components for reference keys via a per-chunk scale $s_d$ interpolated across the spectrum, coupled with a timestep schedule to gradually balance global guidance and fine-grained transfer. Applied to Diffusion Transformers, this method yields stable, semantically guided style transfer without copying the reference content and offers a controllable trade-off between style and content. Overall, the work reveals a principled lever—positional encoding frequency bands—for controlling attention behavior in DiTs and improving practical style-aligned generation.

Abstract

Positional encodings are essential to transformer-based generative models, yet their behavior in multimodal and attention-sharing settings is not fully understood. In this work, we present a principled analysis of Rotary Positional Embeddings (RoPE), showing that RoPE naturally decomposes into frequency components with distinct positional sensitivities. We demonstrate that this frequency structure explains why shared-attention mechanisms, where a target image is generated while attending to tokens from a reference image, can lead to reference copying, in which the model reproduces content from the reference instead of extracting only its stylistic cues. Our analysis reveals that the high-frequency components of RoPE dominate the attention computation, forcing queries to attend mainly to spatially aligned reference tokens and thereby inducing this unintended copying behavior. Building on these insights, we introduce a method for selectively modulating RoPE frequency bands so that attention reflects semantic similarity rather than strict positional alignment. Applied to modern transformer-based diffusion architectures, where all tokens share attention, this modulation restores stable and meaningful shared attention. As a result, it enables effective control over the degree of style transfer versus content copying, yielding a proper style-aligned generation process in which stylistic attributes are transferred without duplicating reference content.

Figures (21)

• Figure 1: Style-aligned image generation aims to produce images with different contents but a consistent style. (a) StyleAligned hertz2023StyleAligned applies shared attention in SDXL, achieving style alignment but introducing artifacts. (b) Flux without attention sharing fails to produce style-aligned image sets. (c) Plain attention sharing in Flux leads to near-identical outputs due to positional bias. (d) Our frequency-aware attention sharing produces style-aligned images while preserving content fidelity to the text prompts.
• Figure 2: RoPE overview. (a) For a token sequence, we illustrate three tokens, $q$, $k_1$, and $k_2$, located at positions $m$, $n_1$, and $n_2$. For each token, we highlight three corresponding chunks: 2-dimensional slices of the embedding on which RoPE applies independent rotations. (b) Assuming the corresponding chunks of $k_1$ and $k_2$ are identical, RoPE rotates them by different angles because their positions $n_1$ and $n_2$ differ. Each chunk is rotated by frequency $\theta_d$, producing position-dependent inner products between $q$ and each key.
• Figure 3: Reference copying. Given a reference image of a bull, we generate a style-aligned image of a giraffe. (a) Without attention sharing, the output is not style-aligned. (b) Applying attention sharing at all denoising timesteps leads to near-exact replication of the reference. (c,d) Restricting attention sharing to later timesteps does not prevent copying. (e) Our method produces a style-aligned result while preserving the target prompt.
• Figure 4: (a) Attention visualization for standard image generation. For each query point (red dots), we show the full attention map, attention using only high-frequency RoPE components, and attention using only low-frequency components (left to right). Retaining only low-frequency components yields more semantically aligned attention. (b) Cross-image attention visualization. Target queries attend to reference image patches; scaling down high-frequency key components similarly makes attention more global.
• Figure 5: Mean attention similarity between two identical vectors as a function of position shift ($\Delta$). Although RoPE naturally reduces attention similarity as the positional shift increases, the rate of decay varies across frequency bands: high-frequency components exhibit a steep drop, mid-frequency components show moderate sensitivity, and low-frequency components are largely insensitive to positional changes.