Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AUGUSTUS: An LLM-Driven Multimodal Agent System with Contextualized User Memory

Jitesh Jain, Shubham Maheshwari, Ning Yu, Wen-mei Hwu, Humphrey Shi

TL;DR

AUGUSTUS addresses the limited multimodal external memory in LLM-driven agents by modeling memory after human cognition, introducing a hierarchical contextual memory of semantic tags and a two-stage CoPe search for concept-driven retrieval. The system operates in an encode–store–retrieve–act loop, leveraging open-source encoders, a Mixtral-based planner, and function calling to autonomously manage memory and tools. Empirical results show CoPe search achieves competitive multimodal retrieval performance on ImageNet while being approximately 3.5× faster than vector-based RAG, and AUGUSTUS outperforms MemGPT on the MSC conversational benchmark, highlighting benefits in personalization and efficiency. This work advances cognition-aligned multimodal agents, enabling robust memory-based reasoning and personalized interactions across image, audio, and video modalities.

Abstract

Riding on the success of LLMs with retrieval-augmented generation (RAG), there has been a growing interest in augmenting agent systems with external memory databases. However, the existing systems focus on storing text information in their memory, ignoring the importance of multimodal signals. Motivated by the multimodal nature of human memory, we present AUGUSTUS, a multimodal agent system aligned with the ideas of human memory in cognitive science. Technically, our system consists of 4 stages connected in a loop: (i) encode: understanding the inputs; (ii) store in memory: saving important information; (iii) retrieve: searching for relevant context from memory; and (iv) act: perform the task. Unlike existing systems that use vector databases, we propose conceptualizing information into semantic tags and associating the tags with their context to store them in a graph-structured multimodal contextual memory for efficient concept-driven retrieval. Our system outperforms the traditional multimodal RAG approach while being 3.5 times faster for ImageNet classification and outperforming MemGPT on the MSC benchmark.

AUGUSTUS: An LLM-Driven Multimodal Agent System with Contextualized User Memory

TL;DR

AUGUSTUS addresses the limited multimodal external memory in LLM-driven agents by modeling memory after human cognition, introducing a hierarchical contextual memory of semantic tags and a two-stage CoPe search for concept-driven retrieval. The system operates in an encode–store–retrieve–act loop, leveraging open-source encoders, a Mixtral-based planner, and function calling to autonomously manage memory and tools. Empirical results show CoPe search achieves competitive multimodal retrieval performance on ImageNet while being approximately 3.5× faster than vector-based RAG, and AUGUSTUS outperforms MemGPT on the MSC conversational benchmark, highlighting benefits in personalization and efficiency. This work advances cognition-aligned multimodal agents, enabling robust memory-based reasoning and personalized interactions across image, audio, and video modalities.

Abstract

Riding on the success of LLMs with retrieval-augmented generation (RAG), there has been a growing interest in augmenting agent systems with external memory databases. However, the existing systems focus on storing text information in their memory, ignoring the importance of multimodal signals. Motivated by the multimodal nature of human memory, we present AUGUSTUS, a multimodal agent system aligned with the ideas of human memory in cognitive science. Technically, our system consists of 4 stages connected in a loop: (i) encode: understanding the inputs; (ii) store in memory: saving important information; (iii) retrieve: searching for relevant context from memory; and (iv) act: perform the task. Unlike existing systems that use vector databases, we propose conceptualizing information into semantic tags and associating the tags with their context to store them in a graph-structured multimodal contextual memory for efficient concept-driven retrieval. Our system outperforms the traditional multimodal RAG approach while being 3.5 times faster for ImageNet classification and outperforming MemGPT on the MSC benchmark.
Paper Structure (27 sections, 6 figures, 4 tables, 1 algorithm)

This paper contains 27 sections, 6 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLM-Driven Agent Systems
  4. Agent Systems with Memory
  5. System Description
  6. Storing Concepts in Contextual Memory
  7. Retrieve from Contextual Memory: CoPe search
  8. Function Calling
  9. Experiments
  10. Concept Retrieval on ImageNet
  11. Conversation Consistency on MSC
  12. Conclusion
  13. Limitations.
  14. Acknowledgements.
  15. Alignment with Human Cognition
  16. ...and 12 more sections

Figures (6)

  • Figure 1: Snapshot of a conversation between a user and AUGUSTUS about the user's pet. Our system calls different functions corresponding to the four stages of operation to engage with the user while thinking internally wei2023chainofthought for careful planning.
  • Figure 2: Organization of Information in the Contextual Memory. AUGUSTUS conceptualizes information into semantic tags connected with the corresponding context node, creating associations among the tags. In the figure, the expanded context node is associated with the concepts listed in its "tags" field. Note that we do not show all associations in the figure due to space constraints (playful-dog-park).
  • Figure 3: Retrieval with CoPe Search. Given a query, AUGUSTUS retrieves relevant concept (pet; dog; corgi). followed by concept-driven context retrieval to send a personalized response to the user (name and image of the pet to help user reminisce).
  • Figure 4: Personalized response to the user. AUGUSTUS retrieves relevant information from the contextual memory by calling the cope_search function to generate an image aligned with the information about dogs from prior user conversations. Note that the image quality depends on our generator, SD-2.1 sd in this case.
  • Figure I: System Prompt for AUGUSTUS.
  • ...and 1 more figures