Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity

Ruijie Quan, Wenguan Wang, Zhibo Tian, Fan Ma, Yi Yang

TL;DR

Psychometry unifies cross-subject fMRI-to-image reconstruction by training a single omnifit model on amalgamated data. It introduces Omni MoE to learn inter-subject commonalities while preserving subject-specific parameters, and Ecphory to augment fMRI embeddings with prestored memories during inference, guiding a diffusion-based generator. Evaluated on NSD, the approach achieves state-of-the-art-like reconstructions, outperforming subject-specific baselines while reducing training complexity and model size. This cross-subject, memory-augmented framework advances practical brain–computer interface capabilities and highlights important considerations for data privacy in shared-model training.

Abstract

Reconstructing the viewed images from human brain activity bridges human and computer vision through the Brain-Computer Interface. The inherent variability in brain function between individuals leads existing literature to focus on acquiring separate models for each individual using their respective brain signal data, ignoring commonalities between these data. In this article, we devise Psychometry, an omnifit model for reconstructing images from functional Magnetic Resonance Imaging (fMRI) obtained from different subjects. Psychometry incorporates an omni mixture-of-experts (Omni MoE) module where all the experts work together to capture the inter-subject commonalities, while each expert associated with subject-specific parameters copes with the individual differences. Moreover, Psychometry is equipped with a retrieval-enhanced inference strategy, termed Ecphory, which aims to enhance the learned fMRI representation via retrieving from prestored subject-specific memories. These designs collectively render Psychometry omnifit and efficient, enabling it to capture both inter-subject commonality and individual specificity across subjects. As a result, the enhanced fMRI representations serve as conditional signals to guide a generation model to reconstruct high-quality and realistic images, establishing Psychometry as state-of-the-art in terms of both high-level and low-level metrics.

Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity

TL;DR

Psychometry unifies cross-subject fMRI-to-image reconstruction by training a single omnifit model on amalgamated data. It introduces Omni MoE to learn inter-subject commonalities while preserving subject-specific parameters, and Ecphory to augment fMRI embeddings with prestored memories during inference, guiding a diffusion-based generator. Evaluated on NSD, the approach achieves state-of-the-art-like reconstructions, outperforming subject-specific baselines while reducing training complexity and model size. This cross-subject, memory-augmented framework advances practical brain–computer interface capabilities and highlights important considerations for data privacy in shared-model training.

Abstract

Reconstructing the viewed images from human brain activity bridges human and computer vision through the Brain-Computer Interface. The inherent variability in brain function between individuals leads existing literature to focus on acquiring separate models for each individual using their respective brain signal data, ignoring commonalities between these data. In this article, we devise Psychometry, an omnifit model for reconstructing images from functional Magnetic Resonance Imaging (fMRI) obtained from different subjects. Psychometry incorporates an omni mixture-of-experts (Omni MoE) module where all the experts work together to capture the inter-subject commonalities, while each expert associated with subject-specific parameters copes with the individual differences. Moreover, Psychometry is equipped with a retrieval-enhanced inference strategy, termed Ecphory, which aims to enhance the learned fMRI representation via retrieving from prestored subject-specific memories. These designs collectively render Psychometry omnifit and efficient, enabling it to capture both inter-subject commonality and individual specificity across subjects. As a result, the enhanced fMRI representations serve as conditional signals to guide a generation model to reconstruct high-quality and realistic images, establishing Psychometry as state-of-the-art in terms of both high-level and low-level metrics.
Paper Structure (18 sections, 1 equation, 7 figures, 2 tables)

This paper contains 18 sections, 1 equation, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Omni MoE for Learning Inter-Subject Commonality and Individual Specificity
  5. Ecphory for Test-Time Reconstruction
  6. Detailed Network Architecture
  7. Experiments
  8. Experimental Setup
  9. Comparison to State-of-the-Arts
  10. Diagnostic Experiment
  11. Conclusion and Discussion
  12. Dataset and fMRI Preprocessing Details
  13. More Reconstruction Results
  14. More Implementation Details
  15. Omni MoE Algorithm
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Current fMRI-to-Image methods (e.g., MindEye scotti2023reconstructing) train subject-specific models (SSM) on their respective data. They suffer obvious performance degradation when utilizing data from all the subjects to train a unified model (UM). Our Psychometry enables consistent performance improvements over MindEye by training one omnifit model on the amalgamated fMRI data.
  • Figure 2: (a) Illustration of the Psychometry framework (§ \ref{['sub_sec:network']}). (b) Omni MoE engages all experts with subject-specific parameters to work together to capture the inter-subject commonality and individual specificity. The detailed illustration of the "split-then-lump" mechanism are presented in Eq. \ref{['eq:2']}-Eq. \ref{['eq:5']}. (c) Ecphory enhances the predicted fMRI embedding by incorporating the retrieved most pertinent "memories", serving as more dependable conditional signals to a pre-trained diffusion model. The reconstruction results for different subjects should align as closely as possible with the visual stimulus, while the inconsistency among the results of different subjects is caused by the individual specificity of each subject's fMRI data. Please refer to § \ref{['sec:method']} for more details.
  • Figure 3: Visual comparison on NSD test. Psychometry trains only one unified model (UM) for once on the amalgamated fMRI data but generates more accurate reconstructions, even compared to two recent methods scotti2023reconstructingtakagi2023high that train subject-specific models (SSM) on their respective data. See § \ref{['subsec_compare_sota']} for more detailed discussion.
  • Figure 4: The comparison scores (Inception and CLIP) and the model parameters vary as the number of experts increases. The size of the marker depends on the model size. See § \ref{['subsec:diaexp']} for details.
  • Figure 5: Splitting weights (Eq. \ref{['eq:2']}) and lumping weights (Eq. \ref{['eq:4']}) across experts for all four subjects. See related analysis in § \ref{['subsec:diaexp']}.
  • ...and 2 more figures