Collaborative Memory: Multi-User Memory Sharing in LLM Agents with Dynamic Access Control

TL;DR

Collaborative Memory enables memory sharing across multiple users and agents under dynamic, asymmetric permissions by modeling user-agent and agent-resource access as time-evolving bipartite graphs $G_{\\mathcal{U}\mathcal{A}}(t)$ and $G_{\\mathcal{A}\mathcal{R}}(t)$. It introduces a two-tier memory system with provenance-coupled fragments and separates read and write policies to enforce privacy while enabling cross-user knowledge transfer; provenance supports retrospective permission checks and auditability. Across three scenarios—fully collaborative, asymmetric, and dynamically evolving—the framework achieves substantial reductions in external knowledge base calls and maintain high accuracy, demonstrating practical benefits for enterprise-like multi-user AI systems. The approach provides a modular, policy-driven substrate compatible with existing memory architectures and offers a clear path toward scalable, trustworthy multi-agent collaboration. The findings highlight the importance of fine-grained, time-aware access control in realizing efficient and safe cross-user reasoning with LLM ensembles.

Abstract

Complex tasks are increasingly delegated to ensembles of specialized LLM-based agents that reason, communicate, and coordinate actions-both among themselves and through interactions with external tools, APIs, and databases. While persistent memory has been shown to enhance single-agent performance, most approaches assume a monolithic, single-user context-overlooking the benefits and challenges of knowledge transfer across users under dynamic, asymmetric permissions. We introduce Collaborative Memory, a framework for multi-user, multi-agent environments with asymmetric, time-evolving access controls encoded as bipartite graphs linking users, agents, and resources. Our system maintains two memory tiers: (1) private memory-private fragments visible only to their originating user; and (2) shared memory-selectively shared fragments. Each fragment carries immutable provenance attributes (contributing agents, accessed resources, and timestamps) to support retrospective permission checks. Granular read policies enforce current user-agent-resource constraints and project existing memory fragments into filtered transformed views. Write policies determine fragment retention and sharing, applying context-aware transformations to update the memory. Both policies may be designed conditioned on system, agent, and user-level information. Our framework enables safe, efficient, and interpretable cross-user knowledge sharing, with provable adherence to asymmetric, time-varying policies and full auditability of memory operations.

Figures (6)

• Figure 1: Illustration of multi-user, multi-agent collaboration. Scenario 1 (top-left): A fully collaborative memory environment where all users share unrestricted access to all agents. Scenario 2 (top-right): An asymmetric collaborative memory setup with heterogeneous privilege levels. Scenario 3 (bottom): Dynamically changing access, where permissions are granted or revoked over time.
• Figure 2: Left: Illustration of the dynamic multi-user environment, where multiple users collaborate with various agents that each have access to different resources. Right: Illustration of Collaborative Memory. User $u_1$ sends a query $q_1$ and receives responses from agents $a_1$ and $a_2$. These responses are passed to the write policies $\pi^\text{write/private}$ and $\pi^\text{write/shared}$, which updates both the private and shared memories. Memory fragments that are not accessible are shown in gray. When user $u_2$ issues another query $q_2$, the agent access resource $r_1$ as well as reading from the collaborative memory, retrieving only the information it is permitted to access.
• Figure 3: Scenario 1 (Fully Collaborative Memory). Performance of MultiHop-Rag under varying degrees of query overlap among five users. Both isolated and collaborative memory exhibit similar accuracy and agent utilization. However, as the number of queries increases (while remains confined to the same six domains), resource usage declines for both approaches—yet collaborative memory consistently achieves a more substantial reduction across all degrees of query overlap.
• Figure 4: Scenario 2 (Asymmetric Collaborative Memory). Resource usage with and without asymmetric collaboration. Even limited cross-user visibility leads to fewer agent or knowledge-base calls than a completely isolated configuration.
• Figure 5: Scenario 3 (Dynamically Evolving Collaborative Memory). System performance over eight time blocks with dynamically changing privileges. Accuracy (top) tracks the available resources, while agent (middle) and resource (bottom) usage also shift in response to access graph updates.