A Framework for Inference Inspired by Human Memory Mechanisms

TL;DR

The paper advances PMI, a cognitive-inspired framework that unifies perception, dual-layer memory (working memory $M_w$ and long-term memory $M_l$), and inference to improve relational reasoning. It introduces external (competitive, top-$k$) and internal (outer-product) memory write channels, and an inference mechanism that retrieves from both memories via content-based addressing and fuses results to form the current understanding $U^t$ with $U^t = \alpha U_w^t + (1-\alpha) U_{wl}^t$. Across tasks—Sort-of-CLEVR, bAbI, equilateral triangle detection, language modeling, and CIFAR-10—the PMI-TR variant consistently improves convergence speed and accuracy, with visualizations indicating memory consolidation as a core contributor to performance. The results suggest memory-augmented architectures anchored in MMS and GWT can enhance both perception and reasoning, with applicability to diverse modalities and backbone models like Transformers and CNNs. Key contributions include a global, dual-level memory framework, differentiable memory precipitation, and empirical evidence of improved relational inference and long-range dependency handling, supported by attention-pattern analyses. $M_w$ and $M_l$ are central to the mechanism, with $M_w \in \mathbb{R}^{N\times D_m}$ and $M_l \in \mathbb{R}^{C\times N\times D_m}$, and the final cognition blends memories via $U^t = \alpha U_w^t + (1-\alpha) U_{wl}^t$.

Abstract

How humans and machines make sense of current inputs for relation reasoning and question-answering while putting the perceived information into context of our past memories, has been a challenging conundrum in cognitive science and artificial intelligence. Inspired by human brain's memory system and cognitive architectures, we propose a PMI framework that consists of perception, memory and inference components. Notably, the memory module comprises working and long-term memory, with the latter endowed with a higher-order structure to retain extensive and complex relational knowledge and experience. Through a differentiable competitive write access, current perceptions update working memory, which is later merged with long-term memory via outer product associations, reducing information conflicts and averting memory overflow. In the inference module, relevant information is retrieved from two separate memory origins and associatively integrated to attain a more comprehensive and precise interpretation of current perceptions. We exploratively apply our PMI to improve prevailing Transformers and CNN models on question-answering tasks like bAbI-20k and Sort-of-CLEVR datasets, as well as detecting equilateral triangles, language modeling and image classification tasks, and in each case, our PMI enhancements consistently outshine their original counterparts significantly. Visualization analyses reveal that relational memory consolidation, along with the interaction and integration of information from diverse memory sources, substantially contributes to the model effectiveness on inference tasks.

Figures (10)

• Figure 1: Framework overview and the process of grasping the current input at calculation step $t$. (a) The memory module consists of WM $M_w$ and LTM $M_l$, each characterized by distinct data structures. (b) WM is updated by current perception via a differentiable and constrained write access, which is then integrated into LTM through outer product association. The inference component retrieves pertinent data from both WM and LTM using content-based addressing MHC and MHSC, respectively. Subsequently, through integration steps, it consolidates info from these sources to generate fresh insight into the input, which is used for next rounds of inference or to directly support the decision-making process.
• Figure 2: Test accuracy vs training iterations for the Sort-of-CLEVR task.
• Figure 3: Detecting Equilateral Triangles. This figure compares the performance of Transformers with our PMI [PMI-TR] against other Transformer baselines.
• Figure 4: Attention patterns between perceptions and memories across different layers of the PMI-TR.
• Figure 5: Test accuracy vs. training iterations for the Sort-of-CLEVR task. Results of the ablation experiments on the memory module's global sharing or persistence.