Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models

Yuto Kojima, Jiarui Xu, Xueyan Zou, Xiaolong Wang

TL;DR

LoRA-TTT tackles distribution shifts in vision-language models by tuning only low-rank adapters in the image encoder at test time, avoiding costly text-prompt tuning. It combines a MEM-based objective with a masked image modeling reconstruction loss, updating just the LoRA parameters for a single test instance and precomputing text features to skip the text encoder during TTT. On two benchmarks spanning 15 datasets, it achieves state-of-the-art zero-shot gains over CLIP baselines and prompt-tuning methods, while reducing memory and runtime overhead. The approach preserves base generalization, improves calibration, and is suitable for memory-constrained edge devices and high-stakes applications, enabling practical deployment of vision-language models.

Abstract

The rapid advancements in vision-language models (VLMs), such as CLIP, have intensified the need to address distribution shifts between training and testing datasets. Although prior Test-Time Training (TTT) techniques for VLMs have demonstrated robust performance, they predominantly rely on tuning text prompts, a process that demands substantial computational resources and is heavily dependent on entropy-based loss. In this paper, we propose LoRA-TTT, a novel TTT method that leverages Low-Rank Adaptation (LoRA), applied exclusively to the image encoder of VLMs. By introducing LoRA and updating only its parameters during test time, our method offers a simple yet effective TTT approach, retaining the model's initial generalization capability while achieving substantial performance gains with minimal memory and runtime overhead. Additionally, we introduce a highly efficient reconstruction loss tailored for TTT. Our method can adapt to diverse domains by combining these two losses, without increasing memory consumption or runtime. Extensive experiments on two benchmarks, covering 15 datasets, demonstrate that our method improves the zero-shot top-1 accuracy of CLIP-ViT-B/16 by an average of 5.79% on the OOD benchmark and 1.36% on the fine-grained benchmark, efficiently surpassing test-time prompt tuning, without relying on any external models or cache.

LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models

TL;DR

LoRA-TTT tackles distribution shifts in vision-language models by tuning only low-rank adapters in the image encoder at test time, avoiding costly text-prompt tuning. It combines a MEM-based objective with a masked image modeling reconstruction loss, updating just the LoRA parameters for a single test instance and precomputing text features to skip the text encoder during TTT. On two benchmarks spanning 15 datasets, it achieves state-of-the-art zero-shot gains over CLIP baselines and prompt-tuning methods, while reducing memory and runtime overhead. The approach preserves base generalization, improves calibration, and is suitable for memory-constrained edge devices and high-stakes applications, enabling practical deployment of vision-language models.

Abstract

The rapid advancements in vision-language models (VLMs), such as CLIP, have intensified the need to address distribution shifts between training and testing datasets. Although prior Test-Time Training (TTT) techniques for VLMs have demonstrated robust performance, they predominantly rely on tuning text prompts, a process that demands substantial computational resources and is heavily dependent on entropy-based loss. In this paper, we propose LoRA-TTT, a novel TTT method that leverages Low-Rank Adaptation (LoRA), applied exclusively to the image encoder of VLMs. By introducing LoRA and updating only its parameters during test time, our method offers a simple yet effective TTT approach, retaining the model's initial generalization capability while achieving substantial performance gains with minimal memory and runtime overhead. Additionally, we introduce a highly efficient reconstruction loss tailored for TTT. Our method can adapt to diverse domains by combining these two losses, without increasing memory consumption or runtime. Extensive experiments on two benchmarks, covering 15 datasets, demonstrate that our method improves the zero-shot top-1 accuracy of CLIP-ViT-B/16 by an average of 5.79% on the OOD benchmark and 1.36% on the fine-grained benchmark, efficiently surpassing test-time prompt tuning, without relying on any external models or cache.

Paper Structure

This paper contains 13 sections, 4 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminaries
  5. LoRA-TTT
  6. Experiments
  7. Experimental setup
  8. Results
  9. Ablation Study
  10. How to apply LoRA for TTT
  11. Masking strategy
  12. Initialization of LoRA weights
  13. Conclusion

Figures (12)

  • Figure S1: Test-time Prompt Tuning shu2022test
  • Figure S4: LoRA-TTT for zero-shot image classification. Our method applies LoRA to the layers of the image encoder in VLMs. LoRA-TTT updates the LoRA parameters using MEM loss and MAE loss, calculated from the top 10% of high-confidence augmented views. This approach allows adaptation to domain shifts with low memory consumption while maintaining generalization ability.
  • Figure S5: CLIP-ViT-B/16
  • Figure S9: Runtime
  • Figure S12: LoRA rank and scale
  • ...and 7 more figures