The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding

TL;DR

Problem: separate semantic and pixel representations hinder unified learning and efficiency. Approach: propose the Prism Hypothesis linking modalities through a shared low-frequency semantic base and higher-frequency detail, and implement Unified Autoencoding with a frequency-band modulator and Residual Split Flow. Contributions: formal framework linking spectral content to modality, novel frequency-based latent factorization, semantic-wise loss, noise-augmented training, and state-of-the-art reconstruction on ImageNet and MS-COCO with diffusion-friendly latents. Impact: offers a practical route to joint understanding and generation with improved consistency, efficiency, and fidelity.

Abstract

Deep representations across modalities are inherently intertwined. In this paper, we systematically analyze the spectral characteristics of various semantic and pixel encoders. Interestingly, our study uncovers a highly inspiring and rarely explored correspondence between an encoder's feature spectrum and its functional role: semantic encoders primarily capture low-frequency components that encode abstract meaning, whereas pixel encoders additionally retain high-frequency information that conveys fine-grained detail. This heuristic finding offers a unifying perspective that ties encoder behavior to its underlying spectral structure. We define it as the Prism Hypothesis, where each data modality can be viewed as a projection of the natural world onto a shared feature spectrum, just like the prism. Building on this insight, we propose Unified Autoencoding (UAE), a model that harmonizes semantic structure and pixel details via an innovative frequency-band modulator, enabling their seamless coexistence. Extensive experiments on ImageNet and MS-COCO benchmarks validate that our UAE effectively unifies semantic abstraction and pixel-level fidelity into a single latent space with state-of-the-art performance.

Figures (5)

• Figure 1: Frequency energy distribution. Normalized energy $e(k)$ across frequency bands for diverse tokenizers. DINOv2 and CLIP focus on low-frequency (semantic) content, while SD-VAE retains more high-frequency energy, capturing finer details.
• Figure 2: Overall architecture of our proposed Unified Autoencoding (UAE). The input image is separately encoded by both a pretrained Semantic Encoder (e.g., DINOv2) and the trainable Unified Encoder. The unified encoder is initialized from the semantic encoder and optimized under two complementary objectives: a semantic-wise loss that aligns low-frequency components decomposed from the semantic encoder’s representations, and a pixel-wise reconstruction loss that enforces visual fidelity via the Pixel Decoder by adaptively dilating the high-frequency components. The decoder employs spectral transform blocks to refine residual-frequency content and produce the reconstructed image. This joint optimization harmonizes semantic structure and pixel detail within a single latent space.
• Figure 3: Retrieval results via frequency filtering. Text–Image retrieval (R@5) remains stable under low-pass filtering but degrades sharply under high-pass filtering, confirming that semantic alignment primarily resides in low-frequency components.
• Figure 4: Qualitative comparison of reconstruction fidelity across autoencoding paradigms. We visualize reconstructed samples from representative methods, including SD-VAE rombach2022ldm, RAE zheng2025rae, and our proposed UAE. Each row corresponds to reconstructions from a fixed source set spanning text, human, object, and artistic domains. UAE produces the most consistent and semantically faithful reconstructions, preserving both high-frequency details (e.g., texture and edge sharpness) and global structure (e.g., layout and color harmony), while reducing the blurring and semantic drift observed in SD-VAE and RAE. (The detail comparisons are denoted in the yellow boxes.)
• Figure 5: t-SNE visualization of semantic embeddings. We compare the feature distributions from the DINOv2 encoder (left) and the band-0 (low-frequency) component of UAE (right). The two plots exhibit similar global structures and class separability, indicating that UAE effectively preserves the semantic organization of the original encoder while introducing a unified latent space that remains compatible with frequency-based factorization.