Data-efficient Large Vision Models through Sequential Autoregression

TL;DR

This work tackles the data inefficiency of large autoregressive vision models by proposing a data-efficient framework that operates on sequential visual tokens. It leverages a VQGAN tokenizer and a causal transformer (LLaMA) to form autoregressive vision models (LVMs) and introduces vision prompting for downstream tasks. The core contributions are data augmentation strategies to balance long-tail, multi-task datasets and knowledge distillation to train compact models without sacrificing performance, with empirical gains shown across segmentation, pose estimation, and deraining. The results demonstrate notable efficiency gains, including an 83.04% ImageNet top-1 accuracy with an 80M parameter model, suggesting that practical, generalist vision models can be built with reduced data and compute while retaining strong cross-task capabilities.

Abstract

Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended to not only comprehend but also seamlessly transit to out-of-domain tasks. However, current endeavors are hamstrung by an over-reliance on colossal models, exemplified by models with upwards of 3B parameters, and the necessity for an extensive corpus of visual data, often comprising a staggering 400B tokens. In this paper, we delve into the development of an efficient, autoregression-based vision model, innovatively architected to operate on a limited dataset. We meticulously demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding during the testing phase. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint, and a marked decrease in training data requirements, thereby paving the way for more sustainable and accessible advancements in the field of generalist vision models. The code is available at https://github.com/ggjy/DeLVM.

Figures (10)

• Figure 1: An overview of our framework. We follow the autoregressive setting proposed in LVM lvm, which encodes input images into a 1D sequence. We further explore the data-efficient LVMs via data augmentation and knowledge distillation strategy.
• Figure 2: Data augmentation yields a similar effect to the introduction of new data. We train a LLaMA-300M model on subsets of SA-1B and augmented COCO-Pose datasets with a fixed 12K training steps. In segmentation task with ample available data, enhanced model performance is observed upon introducing more training data. Conversely, for human pose estimation task with limited data, augmenting the dataset has a comparable impact to the introduction of new training data. The original COCO-Pose dataset contains approximately 0.03 billion tokens.
• Figure 3: Visualization of inference results generated by models trained on datasets with different balancing schemes. The model trained on an unbalanced dataset exhibits biased performance, excelling primarily in the image segmentation task. Rectifying this data imbalance through re-sampling have unfortunately led to a decrease in performance. Conversely, employing augmentation to balance the dataset results in improved visualization outcomes.
• Figure 4: Visualization of the inference results of single-task trained models. The LLaMA-1B teacher model produces the best outcomes, and the LLaMA-300M model trained with KD exhibits greater similarity to the teacher model compared to the model trained from scratch.
• Figure 5: Generated output and the corresponding foreground segmentation results after our post processing. Task prompt contains a sequence of images interleaved with annotations, followed by a test image. Prompts in (a) are with a pink background, in (b) (c) (d) are with a black background.