Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ElasticTok: Adaptive Tokenization for Image and Video

Wilson Yan, Volodymyr Mnih, Aleksandra Faust, Matei Zaharia, Pieter Abbeel, Hao Liu

TL;DR

ElasticTok introduces adaptive tokenization for images and videos by training autoencoders with a left-aligned, mask-based token selection that conditions on prior frames. This approach enables dynamic allocation of tokens at inference time, guided by reconstruction thresholds or target lengths, and scales to long video sequences via Blockwise RingAttention. Empirically, ElasticTok achieves up to 2-5x token savings while preserving reconstruction quality and demonstrates comparable downstream performance to fixed-token baselines in vision-language tasks, with flexible inference strategies and objective- dependent token allocation. The work highlights practical benefits for multimodal models and agents by reducing token budgets and enabling content-aware representations, while suggesting avenues for future improvements in masking and modality generalization.

Abstract

Efficient video tokenization remains a key bottleneck in learning general purpose vision models that are capable of processing long video sequences. Prevailing approaches are restricted to encoding videos to a fixed number of tokens, where too few tokens will result in overly lossy encodings, and too many tokens will result in prohibitively long sequence lengths. In this work, we introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. To enable this in a computationally scalable way, we propose a masking technique that drops a random number of tokens at the end of each frames's token encoding. During inference, ElasticTok can dynamically allocate tokens when needed -- more complex data can leverage more tokens, while simpler data only needs a few tokens. Our empirical evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage, paving the way for future development of more powerful multimodal models, world models, and agents.

ElasticTok: Adaptive Tokenization for Image and Video

TL;DR

ElasticTok introduces adaptive tokenization for images and videos by training autoencoders with a left-aligned, mask-based token selection that conditions on prior frames. This approach enables dynamic allocation of tokens at inference time, guided by reconstruction thresholds or target lengths, and scales to long video sequences via Blockwise RingAttention. Empirically, ElasticTok achieves up to 2-5x token savings while preserving reconstruction quality and demonstrates comparable downstream performance to fixed-token baselines in vision-language tasks, with flexible inference strategies and objective- dependent token allocation. The work highlights practical benefits for multimodal models and agents by reducing token budgets and enabling content-aware representations, while suggesting avenues for future improvements in masking and modality generalization.

Abstract

Efficient video tokenization remains a key bottleneck in learning general purpose vision models that are capable of processing long video sequences. Prevailing approaches are restricted to encoding videos to a fixed number of tokens, where too few tokens will result in overly lossy encodings, and too many tokens will result in prohibitively long sequence lengths. In this work, we introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. To enable this in a computationally scalable way, we propose a masking technique that drops a random number of tokens at the end of each frames's token encoding. During inference, ElasticTok can dynamically allocate tokens when needed -- more complex data can leverage more tokens, while simpler data only needs a few tokens. Our empirical evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage, paving the way for future development of more powerful multimodal models, world models, and agents.

Paper Structure

This paper contains 23 sections, 14 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Blockwise RingAttention
  4. Method
  5. Training
  6. Inference
  7. Experiments
  8. Experimental Setup
  9. Main Results
  10. Downstream Tasks
  11. Inference
  12. Frequency Analysis
  13. Related Work
  14. Discussion and Conclusion
  15. Model configuration
  16. ...and 8 more sections

Figures (14)

  • Figure 1: ElasticTok adaptively represent video based on information available. (Top) Ground-truth video frames. (Middle) Reconstructed frames with varying token usage. (Bottom) The bottom section depicts how ElasticTok dynamically adjusts token allocation over time, with the percentage of tokens used correlating to different content complexities in the video.
  • Figure 2: ElasticTok adaptively encodes image and video to variable length outputs based on the complexity of the input data. Single block uses an Encoder-Decoder pipeline with a sampled latent mask. Multi-block extends this with a Block Causal Mask to handle longer video sequences.
  • Figure 3: ElasticTok adaptively encodes image and video to variable length outputs based on the complexity of the input data (using ElasticTok-VAE). The top rows shows examples of ElasticTok on images. Below shows a video example with: (Top) Ground-truth video frames. (Middle) Reconstructed frames with varying token usage. (Bottom) The bottom section depicts how ElasticTok dynamically adjusts token allocation over time, with the percentage of tokens used correlating to different content complexities in the video.
  • Figure 4: Performance comparison between baseline and ElasticTok-FSQ on ImageNet and Video. The y-axis shows the percentage of samples that satisfy the reconstruction threshold, while the x-axis represents the percentage of tokens used. (Left) On image, ElasticTok achieves a 3.5x and 1.3x efficiency boost at different reconstruction thresholds. (Right) On video, ElasticTok shows a 5x and 2.4x improvement over the baseline, maintaining superior performance while using fewer tokens. Figure \ref{['fig:recon_mse']} in Appendix \ref{['sec:recon_by_mse']} shows reference examples of reconstruction quality for an image at different thresholds.
  • Figure 5: Loss progressively declines as more tokens are used (ElasticTok-FSQ). The top row illustrates the impact on text clarity, while the bottom row shows the effect on image sharpness. The graphs on the right quantify the reconstruction loss relative to token usage percentage, showing a rapid decline as more tokens are consumed.
  • ...and 9 more figures