Implicit Neural Representation Facilitates Unified Universal Vision Encoding

TL;DR

HUVR presents a unified vision encoding framework by introducing an implicit neural representation (INR) hyper-network that jointly supports recognition and reconstruction. The approach introduces patch-wise INR modulation with a global class token and Tiny Tokens (TinToks), augmented by knowledge distillation from pretrained teachers to inject semantic structure into compressed embeddings. Across classification, dense prediction, and image reconstruction, HUVR demonstrates competitive performance against state-of-the-art unsupervised methods and delivers a compact representation that enables generation experiments via diffusion on TinTok latents. While scaling pretraining remains a limitation, the results establish a versatile, theory-grounded path toward universal vision encoding with native reconstruction capabilities. The work thus opens avenues for compressed, unified representations and lays groundwork for future integration with multimodal and generative models.

Abstract

Models for image representation learning are typically designed for either recognition or generation. Various forms of contrastive learning help models learn to convert images to embeddings that are useful for classification, detection, and segmentation. On the other hand, models can be trained to reconstruct images with pixel-wise, perceptual, and adversarial losses in order to learn a latent space that is useful for image generation. We seek to unify these two directions with a first-of-its-kind model that learns representations which are simultaneously useful for recognition and generation. We train our model as a hyper-network for implicit neural representation, which learns to map images to model weights for fast, accurate reconstruction. We further integrate our INR hyper-network with knowledge distillation to improve its generalization and performance. Beyond the novel training design, the model also learns an unprecedented compressed embedding space with outstanding performance for various visual tasks. The complete model competes with state-of-the-art results for image representation learning, while also enabling generative capabilities with its high-quality tiny embeddings. The code is available at https://github.com/tiktok/huvr.

Figures (4)

• Figure 1: (Top) Unified modeling, in terms of both tasks and token dimensions. We propose implicit neural Hyper-networks for Unified Visual Representation, HUVR, with good classification, reconstruction, and segmentation (shown for ViT-B/16 on ImageNet, ImageNet, and ADE20K, respectively). We design our model to generate not only standard-sized tokens, but also Tiny Tokens (TinToks). Here, the tiny embeddings of DINOv3 are generated via principle component analysis (PCA). (Bottom) Reconstruction. We unify recognition and generative task families.
• Figure 2: INR Hyper-Network for unified visual representation. The standard and compressed encodings from our model have powerful recognition and reconstruction capabilities, enabling downstream tasks ranging from classification to image generation.
• Figure 3: Training time improves both classification and reconstruction performance, although longer training yields incrementally smaller gains.
• Figure 4: Generated samples with HUVR embeddings. We use a DiT-XL train on HUVR embeddings with TinTok $d_t=256$. This HUVR is trained with LPIPS and SSIM losses in addition to the pixel-wise and DINOv3 MSE losses. Compared to our Table \ref{['tab:diffusion_results']}, this DiT is trained for 4500k steps instead of 400k steps. For reference, DiT-XL/2 trains their final model for 7000k steps. These results demonstrate many degradations and artifacts, but we hope they work as a proof-of-concept to convey the promise of HUVR for generation. Future work could apply techniques, such as those in RAE zheng2025diffusiontransformersrepresentationautoencoders, to improve the quality.