Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training Compute-Optimal Vision Transformers for Brain Encoding

Sana Ahmadi, Francois Paugam, Tristan Glatard, Pierre Lune Bellec

TL;DR

The results demonstrate that increasing the hidden layer dimensions significantly improves brain encoding performance, as evidenced by higher Pearson correlation coefficients across all subjects, and the effects of data scaling are more significant compared to model scaling in enhancing brain encoding performance.

Abstract

The optimal training of a vision transformer for brain encoding depends on three factors: model size, data size, and computational resources. This study investigates these three pillars, focusing on the effects of data scaling, model scaling, and high-performance computing on brain encoding results. Using VideoGPT to extract efficient spatiotemporal features from videos and training a Ridge model to predict brain activity based on these features, we conducted benchmark experiments with varying data sizes (10k, 100k, 1M, 6M) and different model configurations of GPT-2, including hidden layer dimensions, number of layers, and number of attention heads. We also evaluated the effects of training models with 32-bit vs 16-bit floating point representations. Our results demonstrate that increasing the hidden layer dimensions significantly improves brain encoding performance, as evidenced by higher Pearson correlation coefficients across all subjects. In contrast, the number of attention heads does not have a significant effect on the encoding results. Additionally, increasing the number of layers shows some improvement in brain encoding correlations, but the trend is not as consistent as that observed with hidden layer dimensions. The data scaling results show that larger training datasets lead to improved brain encoding performance, with the highest Pearson correlation coefficients observed for the largest dataset size (6M). These findings highlight that the effects of data scaling are more significant compared to model scaling in enhancing brain encoding performance. Furthermore, we explored the impact of floating-point precision by comparing 32-bit and 16-bit representations. Training with 16-bit precision yielded the same brain encoding accuracy as 32-bit, while reducing training time by 1.17 times, demonstrating its efficiency for high-performance computing tasks.

Training Compute-Optimal Vision Transformers for Brain Encoding

TL;DR

The results demonstrate that increasing the hidden layer dimensions significantly improves brain encoding performance, as evidenced by higher Pearson correlation coefficients across all subjects, and the effects of data scaling are more significant compared to model scaling in enhancing brain encoding performance.

Abstract

The optimal training of a vision transformer for brain encoding depends on three factors: model size, data size, and computational resources. This study investigates these three pillars, focusing on the effects of data scaling, model scaling, and high-performance computing on brain encoding results. Using VideoGPT to extract efficient spatiotemporal features from videos and training a Ridge model to predict brain activity based on these features, we conducted benchmark experiments with varying data sizes (10k, 100k, 1M, 6M) and different model configurations of GPT-2, including hidden layer dimensions, number of layers, and number of attention heads. We also evaluated the effects of training models with 32-bit vs 16-bit floating point representations. Our results demonstrate that increasing the hidden layer dimensions significantly improves brain encoding performance, as evidenced by higher Pearson correlation coefficients across all subjects. In contrast, the number of attention heads does not have a significant effect on the encoding results. Additionally, increasing the number of layers shows some improvement in brain encoding correlations, but the trend is not as consistent as that observed with hidden layer dimensions. The data scaling results show that larger training datasets lead to improved brain encoding performance, with the highest Pearson correlation coefficients observed for the largest dataset size (6M). These findings highlight that the effects of data scaling are more significant compared to model scaling in enhancing brain encoding performance. Furthermore, we explored the impact of floating-point precision by comparing 32-bit and 16-bit representations. Training with 16-bit precision yielded the same brain encoding accuracy as 32-bit, while reducing training time by 1.17 times, demonstrating its efficiency for high-performance computing tasks.

Paper Structure

This paper contains 26 sections, 18 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Materials and Methods
  3. fMRI datasets
  4. Participants
  5. Magnetic resonance imaging
  6. Preprocessing
  7. Brain parcellation
  8. Generating Video Data
  9. Brain encoding
  10. VideoGPT model
  11. Ridge regression
  12. Brain encoding performance and hyper-parameter optimization
  13. Computational environment
  14. Scaling up VideoGPT model
  15. Scaling up GPT in terms of Dataset Size
  16. ...and 11 more sections

Figures (7)

  • Figure 1: Two main steps of brain encoding: Extracting features from movie frames using GPT-2 model and predicting brain response using ridge regression
  • Figure 2: GPT training across different training dataset sizes
  • Figure 3: GPT training across different hidden dimensions
  • Figure 4: GPT training across different number of layers
  • Figure 5: Training GPT with FP32 and FP16 precision
  • ...and 2 more figures