A flexible Bayesian non-parametric mixture model reveals multiple dependencies of swap errors in visual working memory

TL;DR

The paper introduces BNS, a Bayesian non-parametric mixture model for swap errors in visual working memory, where a Gaussian Process prior defines a swap function $f$ that maps distractor displacements to logits governing a circular-response mixture. The model conditionally depends on probe and/or report features, enabling a data-driven, assumption-free inference about how swaps arise from encoding or retrieval processes. Across multiple datasets, BNS uncovers a strong cue-similarity dependence and, in at least one case, a non-monotonic modulation in the report dimension, suggesting encoding-time binding failures alongside retrieval noise. Rigorous model comparison and recovery show that previous interpretations may miss complex cross-dimensional dependencies, and the framework offers a principled tool to guide experimental design and neural-model constraints. While data-hungry, BNS provides a versatile platform for dissecting the multifaceted mechanisms behind swap errors in visual working memory.

Abstract

Human behavioural data in psychophysics has been used to elucidate the underlying mechanisms of many cognitive processes, such as attention, sensorimotor integration, and perceptual decision making. Visual working memory has particularly benefited from this approach: analyses of VWM errors have proven crucial for understanding VWM capacity and coding schemes, in turn constraining neural models of both. One poorly understood class of VWM errors are swap errors, whereby participants recall an uncued item from memory. Swap errors could arise from erroneous memory encoding, noisy storage, or errors at retrieval time - previous research has mostly implicated the latter two. However, these studies made strong a priori assumptions on the detailed mechanisms and/or parametric form of errors contributed by these sources. Here, we pursue a data-driven approach instead, introducing a Bayesian non-parametric mixture model of swap errors (BNS) which provides a flexible descriptive model of swapping behaviour, such that swaps are allowed to depend on both the probed and reported features of every stimulus item. We fit BNS to the trial-by-trial behaviour of human participants and show that it recapitulates the strong dependence of swaps on cue similarity in multiple datasets. Critically, BNS reveals that this dependence coexists with a non-monotonic modulation in the report feature dimension for a random dot motion direction-cued, location-reported dataset. The form of the modulation inferred by BNS opens new questions about the importance of memory encoding in causing swap errors in VWM, a distinct source to the previously suggested binding and cueing errors. Our analyses, combining qualitative comparisons of the highly interpretable BNS parameter structure with rigorous quantitative model comparison and recovery methods, show that previous interpretations of swap errors may have been incomplete.

Figures (7)

• Figure 1: Graphical model of BNS, described in detail in the main text
• Figure 2: Variational approximations of the swap function posterior given synthetic data $q(f)\approx p(f|\mathcal{D}[\theta_\mathcal{M}])$ - A: when fit by the same model structure as the generative model. The distribution $p_f(f)$ used to generate the data is given in green, and is augmented by scaling to show the BNS's versatility in recovering different true underlying swap functions. Note that $f(\boldsymbol{x}) > 1$ is unlikely in real data, as this implies distractors displaced by $\boldsymbol{x}$ in feature space are more likely to be recalled than the cued item. B: when the fitted model has a different structure $\mathcal{M}'\neq\mathcal{M}$ to the generative model. The green posterior is shown as dotted as it does not span the same axes as the infered blue/red posterior, instead spanning the probe (report) axes in the top (bottom) axes. C: when the fitted model has structure $\mathcal{M}'=\texttt{both}$, and the generative model $\mathcal{M}=\texttt{probe}$. Heatmaps show the swap function mean under $q(f)$ in the positive quadrant (excluding the miniumum margin of seperation from the cued item), and inset axes show slices, with darker slices being closer to the cued item in feature space. Bright green indicates large mean of $f$, and dark blue low. In all three cases, the true generative distribution $p(f)$ (of which only the mean is shown) is learned from data (see Figure \ref{['fig:visualise_real_data_fits']} and main text). Left and center: BNS is fitted with a realistic number of trials, and is able to remove most of the variability in the redundant dimension. Right: the same ground truth as in the leftmost plot is refitted with a much larger (10x) amount of data. For ease of viewing, variance of the swap functions are not displayed, nor are the exponentiated swap functions (relevant due to the softmax step in the generative process). BNS's accuracy in capturing the shape of the underlying swap function wavers mostly at unrealistically high values (A, left), or extremely low values which are compressed during exponentiation (C, middle).
• Figure 3: $\Delta\text{BIC}_s(\mathcal{M};\mathcal{M}')$ (average per-trial difference; see main text) for all combinations of model classes. The swap function amplitude refers to modulation of swap dependence in the generating model parameters. Some examples of different amplitudes are shown in Figure \ref{['fig:model_validation']}A. The data generating model $\mathcal{M}$ is denoted by the dots on the abscissa. In most cases, each model is fit with data from 10 'synthetic subjects', each performing 96 trials (see Materials), and scattered coloured dots indicate individual synthetic subjects' $\Delta\text{BIC}_s$ values for that model combination, across many such synthetic datasets and fitted models. Asterisks denote these values fall significantly ($p<0.05$) below 0, indicating successful model recovery for that combination. Where marked, synthetic datasets with 10 synthetic subjects performing 960 tasks were generated and fitted for each synthetic dataset. Black marks represent overall mean for that model class combination.
• Figure 4: $\Delta\text{BIC}^*_s$ (see main text) on various datasets. In all cases, the baseline model $\mathcal{M}'$ is a $\texttt{none}$ (flat) swap function with a von Mises emission and a uniform component ('vM. + unif'), and $\Delta\text{BIC}$ is compared for other swap function forms and for the wrapped stable ('ws') emission distribution. $\ast$ indicates that the spread of this statistic is significantly ($p<0.05$) above 0, i.e. that the denoted model is a better fit to the data than the baseline model.
• Figure 5: Variational posterior mean functions $\mu_q=\mathbb{E}_q[f]$ for one dimensional models in the A: Schneegans2016 separated by role of each feature, and B orientation-cued and C direction-cued McMaster2022 datasets, separated by stimulus strengths (elongation and coherence respectively). We also show the exponent, to show how low evaluations of $f$ are compressed in the softmax function, but error intervals are omitted for clarity.