Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Supervised Pre-training Tasks for an fMRI Time-series Transformer in Autism Detection

Yinchi Zhou, Peiyu Duan, Yuexi Du, Nicha C. Dvornek

TL;DR

A transformer-based self-supervised framework that directly analyzes time-series fMRI data without computing functional connectivity is developed and it is shown that randomly masking entire ROIs gives better model performance than randomly masking time points in the pre-training step.

Abstract

Autism Spectrum Disorder (ASD) is a neurodevelopmental condition that encompasses a wide variety of symptoms and degrees of impairment, which makes the diagnosis and treatment challenging. Functional magnetic resonance imaging (fMRI) has been extensively used to study brain activity in ASD, and machine learning methods have been applied to analyze resting state fMRI (rs-fMRI) data. However, fewer studies have explored the recent transformer-based models on rs-fMRI data. Given the superiority of transformer models in capturing long-range dependencies in sequence data, we have developed a transformer-based self-supervised framework that directly analyzes time-series fMRI data without computing functional connectivity. To address over-fitting in small datasets and enhance the model performance, we propose self-supervised pre-training tasks to reconstruct the randomly masked fMRI time-series data, investigating the effects of various masking strategies. We then finetune the model for the ASD classification task and evaluate it using two public datasets and five-fold cross-validation with different amounts of training data. The experiments show that randomly masking entire ROIs gives better model performance than randomly masking time points in the pre-training step, resulting in an average improvement of 10.8% for AUC and 9.3% for subject accuracy compared with the transformer model trained from scratch across different levels of training data availability. Our code is available on GitHub.

Self-Supervised Pre-training Tasks for an fMRI Time-series Transformer in Autism Detection

TL;DR

A transformer-based self-supervised framework that directly analyzes time-series fMRI data without computing functional connectivity is developed and it is shown that randomly masking entire ROIs gives better model performance than randomly masking time points in the pre-training step.

Abstract

Autism Spectrum Disorder (ASD) is a neurodevelopmental condition that encompasses a wide variety of symptoms and degrees of impairment, which makes the diagnosis and treatment challenging. Functional magnetic resonance imaging (fMRI) has been extensively used to study brain activity in ASD, and machine learning methods have been applied to analyze resting state fMRI (rs-fMRI) data. However, fewer studies have explored the recent transformer-based models on rs-fMRI data. Given the superiority of transformer models in capturing long-range dependencies in sequence data, we have developed a transformer-based self-supervised framework that directly analyzes time-series fMRI data without computing functional connectivity. To address over-fitting in small datasets and enhance the model performance, we propose self-supervised pre-training tasks to reconstruct the randomly masked fMRI time-series data, investigating the effects of various masking strategies. We then finetune the model for the ASD classification task and evaluate it using two public datasets and five-fold cross-validation with different amounts of training data. The experiments show that randomly masking entire ROIs gives better model performance than randomly masking time points in the pre-training step, resulting in an average improvement of 10.8% for AUC and 9.3% for subject accuracy compared with the transformer model trained from scratch across different levels of training data availability. Our code is available on GitHub.
Paper Structure (18 sections, 4 equations, 3 figures, 3 tables)

This paper contains 18 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Self-Supervised Training Framework Overview
  4. Model Architecture
  5. Random Masking in the Pre-training Stage
  6. Experiments
  7. Datasets and Pre-processing
  8. ABIDE I
  9. ACE
  10. Experimental Settings
  11. Implementation Details
  12. Model Evaluation
  13. Results
  14. Pre-training Results with Different Random Masking Strategies
  15. Downstream ASD Classification Results
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Framework overview. A) The proposed self-supervised training workflow consists of a pre-training stage and a fine-tuning stage. The cropped rs-fMRI shown is obtained after data augmentation. B) Three masking strategies that are used in the pre-training tasks. Each row is the time-series data for one ROI.
  • Figure 2: Visualization of example reconstructed sequences from the left-out testing data using different masking strategies.
  • Figure 3: Left: Testing AUC of ASD classification models learned from different percentages of training data using different pre-training masking strategies. Right: Example ROC curves of one test set using 100% training data.