Ensemble Kalman filter for uncertainty in human language comprehension

TL;DR

The paper addresses the gap between deterministic neural language models and human uncertainty in processing, especially under reversal anomalies. It reframes sentence comprehension as a Bayesian inverse problem and applies an ensemble Kalman filter ($\mathrm{EnKF}$) to perform last-layer Bayesian inference in the Sentence Gestalt (SG) model, yielding an approximate posterior $\pi_{\text{post}}(\theta) \propto \exp(-l(\theta))\,\pi_{\text{prior}}(\theta)$. The authors introduce a dropout-deterministic IPS-based sampler to generate posterior samples, demonstrating improved uncertainty calibration over maximum likelihood estimation on a synthetic 10,000-sentence corpus, particularly in RA conditions. This approach enhances cognitive-model realism and provides a scalable framework for uncertainty-aware language processing, with potential extensions to richer semantics and cross-layer Bayesian inference.

Abstract

Artificial neural networks (ANNs) are widely used in modeling sentence processing but often exhibit deterministic behavior, contrasting with human sentence comprehension, which manages uncertainty during ambiguous or unexpected inputs. This is exemplified by reversal anomalies-sentences with unexpected role reversals that challenge syntax and semantics-highlighting the limitations of traditional ANN models, such as the Sentence Gestalt (SG) Model. To address these limitations, we propose a Bayesian framework for sentence comprehension, applying an extension of the ensemble Kalman filter (EnKF) for Bayesian inference to quantify uncertainty. By framing language comprehension as a Bayesian inverse problem, this approach enhances the SG model's ability to reflect human sentence processing with respect to the representation of uncertainty. Numerical experiments and comparisons with maximum likelihood estimation (MLE) demonstrate that Bayesian methods improve uncertainty representation, enabling the model to better approximate human cognitive processing when dealing with linguistic ambiguities.

Figures (9)

• Figure 1: The Sentence Gestalt model
• Figure 2: Last-layer Bayesian inference in SGM using ensemble transform methods
• Figure 3: Training & test loss and accuracy obtained by training SGM with ADAM optimizer.
• Figure 4: Mean activations (with standard deviations) of selected output units for the congruent form of test sentences using the ADAM-trained SG model and the Bayesian SG model (with dropout deterministic sampler). Blue bars indicate the output activations of the selected feature (word) from the ADAM-trained model, while red bars represent the predicted probability of the feature being activated in the Bayesian SG model. Black error bars denote standard deviations.
• Figure 5: Mean activations (with standard deviations) of selected output units for the RA form of test sentences using the ADAM-trained SG model and the Bayesian SG model (with dropout deterministic sampler). Blue bars indicate the output activations of the selected feature (word) from the ADAM-trained model, while red bars represent the predicted probability of the feature being activated in the Bayesian SG model. Black error bars denote standard deviations.