GENESIS: A Generative Model of Episodic-Semantic Interaction

TL;DR

GENESIS presents a unified, generative account of semantic and episodic memory by coupling two limited-capacity VAEs—the Cortical-VAE for semantic processing and the Hippocampal-VAE for episodic encoding—with a Retrieval-Augmented Generator memory. Through capacity-bounded encoding ($C\in\{0,1,2,4,8\}$ nats) and rate–distortion theory, the model reproduces core phenomena across both domains, including statistical learning and generalization, recognition and serial recall, gist-based distortions, and constructive episodic recombination. A key innovation is the bidirectional interaction where semantic representations shape episodic encoding and retrieval, realized via a query–key–value memory framework that supports efficient, content–contextual search. By simulating replay and recombination, GENESIS offers a principled, testable framework for memory as an active, constructive, and resource-bounded process with broad implications for understanding human cognition and its neural substrates.

Abstract

A central challenge in cognitive neuroscience is to explain how semantic and episodic memory, two major forms of declarative memory, typically associated with cortical and hippocampal processing, interact to support learning, recall, and imagination. Despite significant advances, we still lack a unified computational framework that jointly accounts for core empirical phenomena across both semantic and episodic processing domains. Here, we introduce the Generative Episodic-Semantic Integration System (GENESIS), a computational model that formalizes memory as the interaction between two limited-capacity generative systems: a Cortical-VAE, supporting semantic learning and generalization, and a Hippocampal-VAE, supporting episodic encoding and retrieval within a retrieval-augmented generation (RAG) architecture. GENESIS reproduces hallmark behavioral findings, including generalization in semantic memory, recognition, serial recall effects and gist-based distortions in episodic memory, and constructive episodic simulation, while capturing their dynamic interactions. The model elucidates how capacity constraints shape the fidelity and memorability of experiences, how semantic processing introduces systematic distortions in episodic recall, and how episodic replay can recombine previous experiences. Together, these results provide a principled account of memory as an active, constructive, and resource-bounded process. GENESIS thus advances a unified theoretical framework that bridges semantic and episodic memory, offering new insights into the generative foundations of human cognition.

Figures (8)

• Figure 1: Illustration of the Generative Episodic–Semantic Integration System (GENESIS) model. A. The Cortical-VAE (a conditional $\beta$-VAE sohn2015learning) achieves a representative input embedding by merging the Gaussian latent state $z$ with color and digit conditional embeddings, $e_C, e_S$, respectively, and provides the input to either the cortical decoder (blue arrow) or the Hippocampal-VAE (red arrow). B. The Hippocampal-VAE (a $\beta$-VAE kingma2013autohiggins2017beta) generates a compressed representation of the input embedding coming from the Cortical-VAE encoding, and incorporates temporal embeddings, to form an episode-specific key. C. Episodic memory is implemented as a Retrieval-Augmented Generation (RAG) system that memorizes episodes as key-value pairs (or sequences of key-value pairs). Values are item embeddings (possibly compressed by the Cortical-VAE). Keys are a mixture of item embeddings (further compressed by the Hippocampal-VAE) and temporal embeddings. D. Interaction between cortical and hippocampal systems given a query. The query triggers a query-key-value search process that retrieves one or more episodes from Episodic Memory and routes them to the decoder of the Cortical-VAE. See the main text for further explanation.
• Figure 2: Simulation of statistical learning, capacity limitations, and generalization in semantic memory. A. Theoretical rate-distortion curve shannon1959coding. The conditional $\beta$-VAE enables the implementation of the rate-distortion trade-off in the loss terms, as shown in the formula at the top ($e=[e_S, e_C]$ is the set of conditioning embeddings) B. Reconstruction performance of images as a function of Cortical-VAE capacity levels C. Multidimensional Scaling (MDS) of the latent feature maps of images (only red digits for simplicity) processed by the Cortical-VAE under low-capacity (0 nats, left) and high-capacity (8 nats, right). Feature maps are extracted from the first decoder layer. D. Schematic of the Leave-One-(Pair)-Out (LOO) protocol. The Cortical-VAE is trained on an ablated dataset with one digit–color pair (e.g., 5-red) withheld; the missing pair is later generated by combining the corresponding conditional embeddings. E. Simulated exemplars for each left-out digit–color pair. Each grid cell shows examples generated from a model trained with that specific pair ablated. F. Heatmaps indicating the classification accuracy of the generated digit–color pairs. See the main text for further explanation.
• Figure 3: Simulations of recognition memory tasks. A. Encoding in the Cortical-VAE to obtain the item embedding $[z, e_S, e_C]$, where here $z=\mu_\Theta$. B. Memory search mechanism during the "old–new" task. Item embeddings from the target list are compressed by the Hippocampal-VAE and memorized in Episodic Memory. During retrieval, a query–key match is performed (temporal embeddings are omitted here, as they were not used), where the query corresponds to an item that may or may not have appeared in the list. C,D. Reconstruction performance of item embeddings as a function of Hippocampal-VAE capacity levels. E. Query–key match accuracy in the old–new task under the top-1 criterion, showing success rate as a function of list size and Hippocampal-VAE capacity. A success occurs when the query matches the closest key by cosine similarity. F. Query–key match accuracy under the top-3 criterion, where a success occurs if the correct key is retrieved within the three closest matches. See the main text for further explanation.
• Figure 4: Simulations of serial recall tasks. A. A conceptual illustration of the temporal embeddings generation schedule, where the geometric distance between subsequent temporal embeddings decreases as time unfolds. B. Schema describing the usage of the temporal embeddings in the Episodic Memory, as items are learned through time. Internal time representations shape both item-related keys in Episodic Memory and queries at the time of recall. C. Modulation of the recency effect as a function of recall time. The entropy of the recall probability distribution of the list items increases as the recall time is delayed. D. Illustration of the forward effect describing the bias towards positive lags when trying to recall the most remembered item based on a cue. E. Geometric representation of temporal and semantic distances between items in the list. Nodes represent an item presented at a given position $t_0, t_1, \ldots, t_N$. The grey lines represent the path connecting the items in the correct temporal order. F. Serial recall memory search path starting from the first item presented in the list as a cue (the item at $t_0$). Nodes represent the temporal distance between the items in the list. The grey lines present the most likely sequential recall path for a given level of $\phi$. The horizontal nodes list at the bottom is the most likely node sequence recalled. G. Visualizations of the probability of recalling the full sequence in the correct order (top), and the median of the maximum sub-sequence recalled (bottom), as a function of $\phi$. See the main text for further explanation.
• Figure 5: Simulations of gist-based distortion of episodic memory under low-capacity encoding. A. Examples of stimuli reconstructed by the Cortical-VAE from episodic memories encoded with varying capacity levels, showing a progressive shift toward gist-like, prototypical representations as capacity decreases. B. Simulation procedure: image encoding at different capacity levels using the Cortical-VAE encoder, memorization in Episodic Memory, and decoding with the Cortical-VAE decoder. C. Quantification of reconstruction self-similarity using a diversity (Vendi) score computed over ResNet feature maps extracted from images reconstructed at different capacity levels (see Section \ref{['sec:methods']} for details). See the main text for further explanation.