Positional Encoding Field

TL;DR

This paper identifies that Diffusion Transformers rely heavily on positional encodings to enforce spatial structure, revealing patch-level independence that enables geometry-guided editing through PE manipulation. It introduces the Positional Encoding Field (PE-Field), a 3D, depth-aware, hierarchical encoding for tokens that enables volumetric reasoning and sub-patch control within DiTs. By combining token manipulation for view synthesis, depth-aware rotary positional encoding, and multi-level PE hierarchies, the authors formulate NVS-DiT, trained with a rectified-flow objective to fuse observed evidence with generative completion. The approach achieves state-of-the-art single-image novel view synthesis and naturally extends to spatial image editing, illustrating significant potential for geometry-aware generative modeling and controllable content manipulation.

Abstract

Diffusion Transformers (DiTs) have emerged as the dominant architecture for visual generation, powering state-of-the-art image and video models. By representing images as patch tokens with positional encodings (PEs), DiTs combine Transformer scalability with spatial and temporal inductive biases. In this work, we revisit how DiTs organize visual content and discover that patch tokens exhibit a surprising degree of independence: even when PEs are perturbed, DiTs still produce globally coherent outputs, indicating that spatial coherence is primarily governed by PEs. Motivated by this finding, we introduce the Positional Encoding Field (PE-Field), which extends positional encodings from the 2D plane to a structured 3D field. PE-Field incorporates depth-aware encodings for volumetric reasoning and hierarchical encodings for fine-grained sub-patch control, enabling DiTs to model geometry directly in 3D space. Our PE-Field-augmented DiT achieves state-of-the-art performance on single-image novel view synthesis and generalizes to controllable spatial image editing.

Figures (9)

• Figure 1: Illustration of DiT patch-level independence. When positional encodings (PEs) of image tokens or noise tokens are perturbed, the decoded or generated outputs still produce semantically meaningful images. The resulting structures follow the warping imposed by the PE modification, while boundaries between patches remain visually distinct.
• Figure 2: Illustration of our direct novel view synthesis (NVS) Results. We apply 2D positional encodings (PEs) derived from 3D reconstruction and view transformation directly to the source-view image tokens. Using these modified tokens as image conditions in DiT enables direct generation of a relatively accurate novel-view image.
• Figure 3: Illustration of hierarchical RoPE allocation in Flux (24 heads). Each rectangle on the left represents the subvector computed by one head, with colors indicating the RoPE level. Black denotes the original patch-level RoPE ($l=0$), covers a 256 pixels patch. Level $l=1$ corresponds to 64 pixels, and level $l=2$ to 16 pixels. The square on the right represents a patch corresponding to one token, illustrating how different levels of positional encodings map to their respective 2D spatial locations, where $l=2$ corresponds to a $1/16$-sized patch.
• Figure 4: The transformer takes both noise tokens and source-view image tokens. Noise tokens are placed on a 2D grid with depth set to zero, while image tokens are assigned hierarchical PEs according to their projected positions from monocular reconstruction and view transformation, with depth values taken from the reconstruction. Tokens projected outside the grid (e.g., index 6) are discarded, and empty grid locations without image tokens (e.g., index 0) are filled by noise, which is refined to generate plausible content.
• Figure 5: Visualization of novel view synthesis results where the source image (left) is rotated $30^\circ$ to the right. Compared with other methods, our approach achieves accurate viewpoint transformation while preserving consistency with the source image and avoiding noticeable artifacts.