Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Natural Image Autoencoders Compactly Tokenize fMRI Volumes for Long-Range Dynamics Modeling?

Peter Yongho Kim, Juhyeon Park, Jungwoo Park, Jubin Choi, Jungwoo Seo, Jiook Cha, Taesup Moon

Abstract

Modeling long-range spatiotemporal dynamics in functional Magnetic Resonance Imaging (fMRI) remains a key challenge due to the high dimensionality of the four-dimensional signals. Prior voxel-based models, although demonstrating excellent performance and interpretation capabilities, are constrained by prohibitive memory demands and thus can only capture limited temporal windows. To address this, we propose TABLeT (Two-dimensionally Autoencoded Brain Latent Transformer), a novel approach that tokenizes fMRI volumes using a pre-trained 2D natural image autoencoder. Each 3D fMRI volume is compressed into a compact set of continuous tokens, enabling long-sequence modeling with a simple Transformer encoder with limited VRAM. Across large-scale benchmarks including the UK-Biobank (UKB), Human Connectome Project (HCP), and ADHD-200 datasets, TABLeT outperforms existing models in multiple tasks, while demonstrating substantial gains in computational and memory efficiency over the state-of-the-art voxel-based method given the same input. Furthermore, we develop a self-supervised masked token modeling approach to pre-train TABLeT, which improves the model's performance for various downstream tasks. Our findings suggest a promising approach for scalable and interpretable spatiotemporal modeling of brain activity. Our code is available at https://github.com/beotborry/TABLeT.

Can Natural Image Autoencoders Compactly Tokenize fMRI Volumes for Long-Range Dynamics Modeling?

Abstract

Modeling long-range spatiotemporal dynamics in functional Magnetic Resonance Imaging (fMRI) remains a key challenge due to the high dimensionality of the four-dimensional signals. Prior voxel-based models, although demonstrating excellent performance and interpretation capabilities, are constrained by prohibitive memory demands and thus can only capture limited temporal windows. To address this, we propose TABLeT (Two-dimensionally Autoencoded Brain Latent Transformer), a novel approach that tokenizes fMRI volumes using a pre-trained 2D natural image autoencoder. Each 3D fMRI volume is compressed into a compact set of continuous tokens, enabling long-sequence modeling with a simple Transformer encoder with limited VRAM. Across large-scale benchmarks including the UK-Biobank (UKB), Human Connectome Project (HCP), and ADHD-200 datasets, TABLeT outperforms existing models in multiple tasks, while demonstrating substantial gains in computational and memory efficiency over the state-of-the-art voxel-based method given the same input. Furthermore, we develop a self-supervised masked token modeling approach to pre-train TABLeT, which improves the model's performance for various downstream tasks. Our findings suggest a promising approach for scalable and interpretable spatiotemporal modeling of brain activity. Our code is available at https://github.com/beotborry/TABLeT.

Paper Structure

This paper contains 32 sections, 1 equation, 8 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Tokenization of fMRI with 2D Natural Image Autoencoder
  5. TABLeT Model Architecture
  6. Self-supervised Pre-training with Masked Token Modeling
  7. Experimental Results
  8. Experimental Setting
  9. Main Results
  10. Effect of Pre-training on Downstream Tasks
  11. Comparison of 2D Natural Image DCAE and 3D fMRI-trained DCAE
  12. Memory and Computational Efficiency
  13. Additional Ablation Studies
  14. Interpretation Results
  15. Conclusion & Limitations
  16. ...and 17 more sections

Figures (8)

  • Figure 1: We show that a 2D natural image autoencoder can be transferred well to 4D fMRI data (left). Building on this, we propose to tokenize a single 3D volume of fMRI only into 27 tokens, enabling longer spatiotemporal modeling with a simple Transformer architecture (right).
  • Figure 2: In TABLeT, each frame of the fMRI timeseries is tokenized by a 2D autoencoder, and the tokens are processed by a Transformer.
  • Figure 3: Visualization of reconstructions from 3D and 2D DCAE.
  • Figure 4: Information preservation of 3D and 2D DCAE.
  • Figure 5: Comparison of (a) memory and (b) training time, between TABLeT and SwiFT.
  • ...and 3 more figures