Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More

TL;DR

This work reveals Patchification Scaling Laws: reducing patch size in ViT-like encoders consistently improves performance across vision tasks, up to pixel-tokenization at 1x1 patches. By evaluating ViT and Adventurer on classification, segmentation, and detection, the authors show that a non-compressive, pixel-level representation can yield higher accuracy (e.g., 84.6% on ImageNet-1k with a 50,176-token sequence) and reduce reliance on task-specific decoders for dense prediction. The findings challenge the necessity of patch-based compression under abundant compute and highlight a path toward encoder-only, pixel-level vision models. The study also provides practical insights into scaling trade-offs between patch size, model size, and input resolution, with implications for future visual foundation models.

Abstract

Since the introduction of Vision Transformer (ViT), patchification has long been regarded as a de facto image tokenization approach for plain visual architectures. By compressing the spatial size of images, this approach can effectively shorten the token sequence and reduce the computational cost of ViT-like plain architectures. In this work, we aim to thoroughly examine the information loss caused by this patchification-based compressive encoding paradigm and how it affects visual understanding. We conduct extensive patch size scaling experiments and excitedly observe an intriguing scaling law in patchification: the models can consistently benefit from decreased patch sizes and attain improved predictive performance, until it reaches the minimum patch size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable across different vision tasks, various input scales, and diverse architectures such as ViT and the recent Mamba models. Moreover, as a by-product, we discover that with smaller patches, task-specific decoder heads become less critical for dense prediction. In the experiments, we successfully scale up the visual sequence to an exceptional length of 50,176 tokens, achieving a competitive test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We hope this study can provide insights and theoretical foundations for future works of building non-compressive vision models. Code is available at https://github.com/wangf3014/Patch_Scaling.

Figures (4)

• Figure 1: Patchification Scaling Laws. We observe a smooth and consistent decrease in test loss across different vision tasks, input resolutions, and model architectures when reducing the patch size. The performance gains remain considerably significant even when scaling down the patch size to 1$\times$1. In all sub-figures, both $x$ and $y$ axes are in log scale. CLS denotes ImageNet-1k classification.
• Figure 2: Decoder's impact on semantic segmentation. We train a semantic segmentation model with the same backbone but different decoder heads: an UperNet with 13M parameters and a simple linear layer with 0.2M parameters. We observe that as patch size decreases, the impact of the decoder head diminishes.
• Figure 3: Patch size scaling vs. parameter scaling. Given an Adventurer-Base with 224$\times$224-resolution inputs, we scale up the model along two dimensions respectively. The model struggles to achieve further accuracy improvements beyond $\sim$760M parameters, whereas scaling down the patch size continues to show a consistent upward trend in performance.
• Figure 4: Input size scaling with fixed sequence length. We fix the ratio of $image\_size / patch\_size$ and scale up the input size for ImageNet classification. As shown, when the input size is scaled beyond its original resolutions (e.g., typically 460 for ImageNet), further interpolating the input images does not yield additional accuracy gains. Instead, it leads to a rapid increase in patchification parameters, resulting in training instability that ultimately harms performance.