INTERN: A New Learning Paradigm Towards General Vision

TL;DR

INTERN tackles the data bottleneck in vision by proposing a three-stage upstream pretraining that evolves from an amateur to a generalist, coupled with a downstream adaptation framework. It introduces GV-D/GV-A/GV-B as an ecological backbone and MetaNet as a billion-level hybrid architecture, optimized via unified architecture search. The framework demonstrates strong data efficiency, with 10% downstream data often outperforming full-data baselines, and shows extensibility by seamlessly adding more task-specific experts. Together with a comprehensive benchmark and data ecosystem, INTERN advances practical general-vision capabilities with significant potential to reduce data requirements and accelerate AI deployment across diverse domains.

Abstract

Enormous waves of technological innovations over the past several years, marked by the advances in AI technologies, are profoundly reshaping the industry and the society. However, down the road, a key challenge awaits us, that is, our capability of meeting rapidly-growing scenario-specific demands is severely limited by the cost of acquiring a commensurate amount of training data. This difficult situation is in essence due to limitations of the mainstream learning paradigm: we need to train a new model for each new scenario, based on a large quantity of well-annotated data and commonly from scratch. In tackling this fundamental problem, we move beyond and develop a new learning paradigm named INTERN. By learning with supervisory signals from multiple sources in multiple stages, the model being trained will develop strong generalizability. We evaluate our model on 26 well-known datasets that cover four categories of tasks in computer vision. In most cases, our models, adapted with only 10% of the training data in the target domain, outperform the counterparts trained with the full set of data, often by a significant margin. This is an important step towards a promising prospect where such a model with general vision capability can dramatically reduce our reliance on data, thus expediting the adoption of AI technologies. Furthermore, revolving around our new paradigm, we also introduce a new data system, a new architecture, and a new benchmark, which, together, form a general vision ecosystem to support its future development in an open and inclusive manner. See project website at https://opengvlab.shlab.org.cn .

Figures (12)

• Figure 1: Comparison of transfer learning performance on diverse tasks. Our largest pretrained model, Up-G MN-B15, with 90% fewer downstream data, surpasses the best publicly available pretrained model (CLIP-R50$\times$16) on most tasks. All results are obtained with backbone parameters fixed during downstream training. Note that the last three datasets in the pink background use the $y$-axis on the right, and the lower bar means the better performance.
• Figure 2: Overview of INTERN. A complete flow of learning and evaluating a general vision model consists of three fundamental bases (i.e. GV-Dataset, GV-Architecture, and GV-Benchmark), a three-stage upstream pretraining scheme (i.e. Amateur, Expert, and Generalist), and a downstream adaptation algorithm that transfers the up-pretrained models to various downstream tasks in the benchmark. It shows that a general model (e.g. Generalist) with a gradual-learning process behaves stronger generalizability even with unseen tasks (shown in a red question mark).
• Figure 3: Searched MetaNet architecture. Conv and Trans represent convolution and transformer blocks respectively. C and S refer to the output channel number and stride of each stage.
• Figure 4: Framework of Upstream-Amateur. The Up-A stage has two pretraining phases: Upstream-Amateur for Global Representation (Up-A-G) and Upstream-Amateur for Local Representation (Up-A-L). Up-A-G (left) uses group-supervision functions for learning from richer supervision. $\{z^I,z^I{'}\}$ and $\{z^T,z^T{'}\}$ are embedding features of the augmented images and texts separately. To improve the performance of Up-A-G on dense prediction CV tasks, in the phase of Up-A-L (right), the well-trained vision-language models are adapted by the local self-supervision learning method. $\{C2 \small{\sim} C5\}$ and $\{C2' \small{\sim} C5'\}$ denote the feature maps of the augmented images from the frozen backbone network, in which the numeric subscripts represent the level of features. "Proj." and "Pred." are abbreviations of the projector and predictor. The online and target network are derived from BYOL grill2020bootstrap, which can help prevent models from collapsing.
• Figure 5: Overview of our Up-E stage architecture. The diagrams indicate our Up-E (C) , Up-E (D), and Up-E (S) expert models. For Up-E (C), we put fully connected heads on top of the shared backbones for predicting the category. For Up-E (D), we first use a shared FPN to facilitate sharing between detection tasks, and then several Faster R-CNN heads are placed upon the FPN network for detecting objects. We use Deeplabv3 heads for our segmentation expert Up-E (S).