Pre-training Vision Transformers with Formula-driven Supervised Learning

TL;DR

This work investigates pre-training Vision Transformers using formula-driven supervised learning (FDSL) as a privacy- and label-free alternative to real-image datasets. By introducing RCDB and ExFractalDB, it demonstrates that contour-based synthetic images can yield ViT representations that fine-tune competitively on ImageNet-1k, even matching ImageNet-21k pre-training and approaching JFT-300M performance (e.g., ExFractalDB-21k at 83.8% top-1 vs 84.1% for JFT-300M). The study validates two hypotheses: (1) object contours are crucial for effective FDSL pre-training, and (2) increasing the complexity/number of parameters in the synthetic generation improves downstream accuracy. The findings suggest synthetic, formula-generated data can scale to large-class regimes (21k/50k) and approach state-of-the-art results while avoiding real-image biases, with practical implications for ethical, scalable foundation-model pre-training.

Abstract

In the present work, we show that the performance of formula-driven supervised learning (FDSL) can match or even exceed that of ImageNet-21k and can approach that of the JFT-300M dataset without the use of real images, human supervision, or self-supervision during the pre-training of vision transformers (ViTs). For example, ViT-Base pre-trained on ImageNet-21k and JFT-300M showed 83.0 and 84.1% top-1 accuracy when fine-tuned on ImageNet-1k, and FDSL showed 83.8% top-1 accuracy when pre-trained under comparable conditions (hyperparameters and number of epochs). Especially, the ExFractalDB-21k pre-training was calculated with x14.2 fewer images compared with JFT-300M. Images generated by formulas avoid privacy and copyright issues, labeling costs and errors, and biases that real images suffer from, and thus have tremendous potential for pre-training general models. To understand the performance of the synthetic images, we tested two hypotheses, namely (i) object contours are what matter in FDSL datasets and (ii) an increased number of parameters for label creation improves performance in FDSL pre-training. To test the former hypothesis, we constructed a dataset that consisted of simple object contour combinations. We found that this dataset matched the performance of fractal databases. For the latter hypothesis, we found that increasing the difficulty of the pre-training task generally leads to better fine-tuning accuracy.

Figures (10)

• Figure 1: We have found that ViTs can be successfully pre-trained without real images, human supervision, or self-supervision, and can exceed the accuracy of ImageNet-21k pre-training when fine-tuned on ImageNet-1k. We constructed a new dataset, Radial Contour DataBase (RCDB), based on the assumption that contours are what matter for the pre-training of ViTs. Even though the ImageNet-21k pre-trained models have an overwhelming advantage in terms of ImageNet-1k fine-tuning because the pre-trained models have already looked at and trained on all images of the fine-tuning dataset, the Extended Fractal DataBase (ExFractalDB-21k) pre-trained model surpassed the ImageNet-21k pre-trained model. The ExFractalDB-21k pre-trained model has been improved from the original FractalDB pre-trained model based on two hypotheses presented in this paper. Moreover, RCDB also exceeded the performance of ImageNet-21k pre-training, while consisting only of contours.
• Figure 2: Fractal images and attention maps on object contours. The first and second columns respectively show original images on FractalDB and attention maps in self-attention on the model. According to these figures, ViTs apparently focus on outer contours in fractal images while acquiring visual representation in pre-training phase. We developed hypothesis 1 in this paper with a preliminary study.
• Figure 3: Procedure for generating radial contours $\mathcal{R}$. An example with $n=3$ vertices and $N=5$ polygons is shown.
• Figure 4: Classes and instances in FractalDB and ExFractalDB.
• Figure 5: Classes and instances in RCDB with only #vertices and parameter set $\eta$.