Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoding Predictive Inference in Visual Language Processing via Spatiotemporal Neural Coherence

Sean C. Borneman, Julia Krebs, Ronnie B. Wilbur, Evie A. Malaia

TL;DR

This work investigates how Deaf signers perform predictive inference during visual language processing by fusing EEG with optical-flow motion features to quantify neural tracking of sign-language structure. A coherence-based, multimodal framework across frequencies and brain regions reveals left-hemispheric and frontal low-frequency involvement, with age-related enhancements in multi-timescale predictive signals. Entropy-based feature selection and two modeling pipelines demonstrate robust age prediction and high accuracy in distinguishing structured from reversed, linguistically disrupted stimuli, supporting predictive coding in visual language and enabling reverse-engineering of internal generative models from brain–stimulus coupling. The approach offers a general, interpretable template for studying experience-driven neural adaptation in complex natural tasks and has potential implications for AI system design and cognitive aging biomarkers.

Abstract

Human language processing relies on the brain's capacity for predictive inference. We present a machine learning framework for decoding neural (EEG) responses to dynamic visual language stimuli in Deaf signers. Using coherence between neural signals and optical flow-derived motion features, we construct spatiotemporal representations of predictive neural dynamics. Through entropy-based feature selection, we identify frequency-specific neural signatures that differentiate interpretable linguistic input from linguistically disrupted (time-reversed) stimuli. Our results reveal distributed left-hemispheric and frontal low-frequency coherence as key features in language comprehension, with experience-dependent neural signatures correlating with age. This work demonstrates a novel multimodal approach for probing experience-driven generative models of perception in the brain.

Decoding Predictive Inference in Visual Language Processing via Spatiotemporal Neural Coherence

TL;DR

This work investigates how Deaf signers perform predictive inference during visual language processing by fusing EEG with optical-flow motion features to quantify neural tracking of sign-language structure. A coherence-based, multimodal framework across frequencies and brain regions reveals left-hemispheric and frontal low-frequency involvement, with age-related enhancements in multi-timescale predictive signals. Entropy-based feature selection and two modeling pipelines demonstrate robust age prediction and high accuracy in distinguishing structured from reversed, linguistically disrupted stimuli, supporting predictive coding in visual language and enabling reverse-engineering of internal generative models from brain–stimulus coupling. The approach offers a general, interpretable template for studying experience-driven neural adaptation in complex natural tasks and has potential implications for AI system design and cognitive aging biomarkers.

Abstract

Human language processing relies on the brain's capacity for predictive inference. We present a machine learning framework for decoding neural (EEG) responses to dynamic visual language stimuli in Deaf signers. Using coherence between neural signals and optical flow-derived motion features, we construct spatiotemporal representations of predictive neural dynamics. Through entropy-based feature selection, we identify frequency-specific neural signatures that differentiate interpretable linguistic input from linguistically disrupted (time-reversed) stimuli. Our results reveal distributed left-hemispheric and frontal low-frequency coherence as key features in language comprehension, with experience-dependent neural signatures correlating with age. This work demonstrates a novel multimodal approach for probing experience-driven generative models of perception in the brain.
Paper Structure (14 sections, 5 equations, 2 figures)

This paper contains 14 sections, 5 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Participants & Stimuli
  5. EEG Recording & Preprocessing
  6. Optical Flow & Neural Coherence Analysis
  7. Feature Selection & Machine Learning
  8. Results
  9. Age Prediction from Neural Coherence
  10. Stimulus Classification Performance
  11. Experience-Dependent Neural Signatures
  12. Discussion
  13. Limitations & Future Work
  14. Broader Impact

Figures (2)

  • Figure 1: Age prediction performance. Boxplots show cross-validated MSE distributions (lower = better). Feature selection improved consistency for tree-based models while maintaining performance for regularized approaches. Baseline (mean age prediction): MSE = 151 years$^2$; best models achieved 85-100 years$^2$ (RMSE 9-10 years).
  • Figure 2: Age-related changes in predictive dynamics. (a) Enhanced low-frequency coherence suggests improved higher-order predictions with experience. (b) Increased delays to unstructured input reflect greater reliance on learned generative models that incur larger prediction error costs when violated.