TULIP: Token-length Upgraded CLIP

TL;DR

The paper tackles the fixed 77-token input limit of CLIP-like vision-language models by introducing TULIP, which employs relative positional encodings (RoPE) to enable arbitrary caption lengths. It presents a two-stage training procedure: first, relative position distillation from the CLIP teacher to a relative-encoding student to preserve short-caption alignment, and second, relative position expansion with NTK-aware RoPE scaling to incorporate longer captions. Key contributions include the first CLIP-like system with relative encodings for long captions, a practical two-phase adaptation framework, and strong improvements on long-caption cross-modal retrieval and text-to-image generation, along with a new long-caption benchmark (Long-DCI). The approach demonstrates robust cross-modal alignment across datasets and backbones, offering a plug-and-play pathway to extend context length without full retraining. This work significantly advances the handling of dense, lengthy textual descriptions in vision-language tasks, enabling more faithful image–text understanding and generation in real-world scenarios.

Abstract

We address the challenge of representing long captions in vision-language models, such as CLIP. By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions. Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to a fixed new token length. Instead, we propose a generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by a training procedure that (i) distills the original CLIP text encoder into an encoder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such as retrieval and text-to-image generation. The code repository is available at https://github.com/ivonajdenkoska/tulip.

Figures (17)

• Figure 1: Swapping the Positional Encoding. We update CLIP models by replacing the absolute positional encoding with relative positional encoding in each transformer block. This modification allows for long caption understanding and better modeling of pairwise token dependencies.
• Figure 2: TULIP training procedure. First, we perform relative position adaptation by distilling the knowledge of the CLIP text encoder into a student text encoder initialized with relative position encodings. This stage uses the first 77 tokens of a long caption (the gray block). The second stage is the relative position expansion, where we fine-tune the distilled text encoder with captions longer than 77 tokens (the combined gray and yellow blocks), together with the vision encoder.
• Figure 3: Text-to-Image Generation results. We replace the text encoder of SDXL with our own TULIP model. We observe improvements in capturing nuanced details, compared to T5-based models such as PIXART-Alpha and ELLA, as well as CLIP-ViT-L-14 and Long-CLIP. Note that the // marks the 77-token boundary in the caption. The words in green indicate visual concepts that are correctly generated by SDXL + TULIP and are missed by the baselines.
• Figure 4: Impact of the sequence length on cross-modal retrieval tasks. We observe general improvement in performance with increased sequence length, particularly from 77 to 154 tokens, across all datasets and tasks.
• Figure 5: Attention Spread Visualization comparing (a) LongClip and (b) TULIP. Our model achieves uniform attention across tokens, demonstrating superior capabilities in parsing and segmenting longer texts with precision.