Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Grounded Memory System For Smart Personal Assistants

Felix Ocker, Jörg Deigmöller, Pavel Smirnov, Julian Eggert

TL;DR

This work tackles the need for robust, long-term memory in personal assistants by proposing a grounded memory system built around three pillars: Grounded Perception, Memory Graphs, and Agentic Retrieval. It combines Vision-Language Model perception with a structured knowledge graph augmented by vector embeddings, and employs Retrieval Augmented Generation through semantic search, graph expansion, and text2cypher querying to ground answers in relational memory. The authors introduce a MemoryNote schema that maps observations into Image notes linked to Agents, Objects, and Actions, enabling temporally ordered, interpretable memory. A working prototype using Neo4j, a vector store, and multimodal capture demonstrates end-to-end ingestion and retrieval, highlighting potential applications in robotics and dementia support. The paper outlines future work on scaling, longer multimodal sequences, disambiguation, and multimodal extensions to enhance context-aware reasoning in autonomous agents.

Abstract

A wide variety of agentic AI applications - ranging from cognitive assistants for dementia patients to robotics - demand a robust memory system grounded in reality. In this paper, we propose such a memory system consisting of three components. First, we combine Vision Language Models for image captioning and entity disambiguation with Large Language Models for consistent information extraction during perception. Second, the extracted information is represented in a memory consisting of a knowledge graph enhanced by vector embeddings to efficiently manage relational information. Third, we combine semantic search and graph query generation for question answering via Retrieval Augmented Generation. We illustrate the system's working and potential using a real-world example.

A Grounded Memory System For Smart Personal Assistants

TL;DR

This work tackles the need for robust, long-term memory in personal assistants by proposing a grounded memory system built around three pillars: Grounded Perception, Memory Graphs, and Agentic Retrieval. It combines Vision-Language Model perception with a structured knowledge graph augmented by vector embeddings, and employs Retrieval Augmented Generation through semantic search, graph expansion, and text2cypher querying to ground answers in relational memory. The authors introduce a MemoryNote schema that maps observations into Image notes linked to Agents, Objects, and Actions, enabling temporally ordered, interpretable memory. A working prototype using Neo4j, a vector store, and multimodal capture demonstrates end-to-end ingestion and retrieval, highlighting potential applications in robotics and dementia support. The paper outlines future work on scaling, longer multimodal sequences, disambiguation, and multimodal extensions to enhance context-aware reasoning in autonomous agents.

Abstract

A wide variety of agentic AI applications - ranging from cognitive assistants for dementia patients to robotics - demand a robust memory system grounded in reality. In this paper, we propose such a memory system consisting of three components. First, we combine Vision Language Models for image captioning and entity disambiguation with Large Language Models for consistent information extraction during perception. Second, the extracted information is represented in a memory consisting of a knowledge graph enhanced by vector embeddings to efficiently manage relational information. Third, we combine semantic search and graph query generation for question answering via Retrieval Augmented Generation. We illustrate the system's working and potential using a real-world example.
Paper Structure (12 sections, 5 figures, 1 table)

This paper contains 12 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Grounded Perception
  4. Memory Graphs
  5. Agentic Retrieval
  6. Grounded Memory System Architecture
  7. Representing Memory Notes
  8. Perception
  9. Knowledge Graph and Vector Store Population
  10. Agentic Retrieval for Question Answering
  11. Implementation Details
  12. Summary and Outlook

Figures (5)

  • Figure 1: Memory system architecture overview.
  • Figure 2: Schema for representing images and action patterns.
  • Figure 3: Three-step prompt applied to an image sequence. Each processing window consists of multiple consecutive frames, analyzed together. The first and last frame overlap with adjacent windows to ensure continuity. Captions for the sequence are appended to the last frame.
  • Figure 4: Knowledge graph (left) and image note (right). Sand-colored nodes represent sequential video frames, green nodes denote detected object instances, the pink node corresponds to the agent, and yellow nodes represent actions. Entities are linked across frames and via action patterns.
  • Figure 5: Chat frontend for interacting with the memory system.