Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neuro-Symbolic Decoding of Neural Activity

Yanchen Wang, Joy Hsu, Ehsan Adeli, Jiajun Wu

TL;DR

It is demonstrated that incorporating structural priors into the decoding process significantly improves both decoding accuracy over precise queries, and notably, generalization to unseen queries at test time.

Abstract

We propose NEURONA, a neuro-symbolic framework for fMRI decoding and concept grounding in neural activity. Leveraging image- and video-based fMRI question-answering datasets, NEURONA learns to decode interacting concepts from visual stimuli based on patterns of fMRI responses, integrating symbolic reasoning and compositional execution with fMRI grounding across brain regions. We demonstrate that incorporating structural priors (e.g., compositional predicate-argument dependencies between concepts) into the decoding process significantly improves both decoding accuracy over precise queries, and notably, generalization to unseen queries at test time. With NEURONA, we highlight neuro-symbolic frameworks as promising tools for understanding neural activity.

Neuro-Symbolic Decoding of Neural Activity

TL;DR

It is demonstrated that incorporating structural priors into the decoding process significantly improves both decoding accuracy over precise queries, and notably, generalization to unseen queries at test time.

Abstract

We propose NEURONA, a neuro-symbolic framework for fMRI decoding and concept grounding in neural activity. Leveraging image- and video-based fMRI question-answering datasets, NEURONA learns to decode interacting concepts from visual stimuli based on patterns of fMRI responses, integrating symbolic reasoning and compositional execution with fMRI grounding across brain regions. We demonstrate that incorporating structural priors (e.g., compositional predicate-argument dependencies between concepts) into the decoding process significantly improves both decoding accuracy over precise queries, and notably, generalization to unseen queries at test time. With NEURONA, we highlight neuro-symbolic frameworks as promising tools for understanding neural activity.
Paper Structure (70 sections, 11 equations, 9 figures, 20 tables)

This paper contains 70 sections, 11 equations, 9 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Visual decoding from fMRI.
  4. Concept grounding.
  5. fMRI-question answering.
  6. Method
  7. Neuro-symbolic framework
  8. Grounding concepts onto neural activity
  9. Unary concepts.
  10. Relational concepts.
  11. From groundings to answers.
  12. Testing hypotheses of grounding structures
  13. H1: Single-region localized.
  14. H2: Multi-region co-activation.
  15. H3: Subject-guided multi-region.
  16. ...and 55 more sections

Figures (9)

  • Figure 1: NEURONA is a neuro-symbolic framework for neural decoding that parses each query into a symbolic expression and maps the accompanying fMRI recording into candidate parcel-level embeddings. It grounds concepts in the expression to these candidate parcels with learned linear concept modules, optionally guided by predicate-argument structure, and composes the grounded scores to answer the question. Supervision is provided only by the final answer, which enables learning of intermediate groundings.
  • Figure 2: We include example queries and dataset distribution overviews for BOLD5000-QA and CNeuroMod-QA; both datasets span diverse queries and tasks.
  • Figure 3: We show examples of learned grounding from NEURONA. On both BOLD5000-QA and CNeuroMod-QA, we see that predicate concepts ground to regions that their constituent objects arguments are grounded to, following hierarchical predicate-argument structure.
  • Figure 4: Statistical tests evaluating the effects of the five hypotheses across subjects in BOLD5000.
  • Figure 5: Statistical tests evaluating the effects of the five hypotheses across subjects in CNeuroMod.
  • ...and 4 more figures