Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cost and accuracy of long-term graph memory in distributed LLM-based multi-agent systems

Benedict Wolff, Jacopo Bennati

TL;DR

This paper tackles the challenge of evaluating long-term memory in distributed LLM-based multi-agent systems under network constraints. It introduces a flexible testbed and directly compares two memory backends—mem0 (vector-based) and Graphiti (graph-based)—using the LOCOMO long-context benchmark across unconstrained and constrained network conditions. The authors find that mem0 offers a substantially lower cost and faster memory operations, while accuracy differences with Graphiti are not statistically significant, leading to mem0 as the Pareto-efficient choice in terms of cost-accuracy balance. The work provides a foundation for cost-aware memory design in DMAS and points to future research in autonomous networking, dynamic deployment, and broader memory architectures.

Abstract

Distributed multi-agent systems use large language models to enable collaborative intelligence while preserving privacy, yet systematic evaluations of long-term memory under network constraints remain limited. This study presents a flexible testbed comparing mem0, a vector-based memory framework, and Graphiti, a graph-based knowledge graph, using the LOCOMO long-context benchmark. Experiments were conducted under unconstrained and constrained network conditions, measuring computational, financial, and accuracy metrics. Results indicate that mem0 significantly outperforms Graphiti in efficiency, with faster loading times, lower resource consumption, and minimal network overhead, while accuracy differences are not statistically significant. Applying a statistical pareto efficiency framework, mem0 is identified as the optimal choice that balances cost and accuracy in DMAS.

Cost and accuracy of long-term graph memory in distributed LLM-based multi-agent systems

TL;DR

This paper tackles the challenge of evaluating long-term memory in distributed LLM-based multi-agent systems under network constraints. It introduces a flexible testbed and directly compares two memory backends—mem0 (vector-based) and Graphiti (graph-based)—using the LOCOMO long-context benchmark across unconstrained and constrained network conditions. The authors find that mem0 offers a substantially lower cost and faster memory operations, while accuracy differences with Graphiti are not statistically significant, leading to mem0 as the Pareto-efficient choice in terms of cost-accuracy balance. The work provides a foundation for cost-aware memory design in DMAS and points to future research in autonomous networking, dynamic deployment, and broader memory architectures.

Abstract

Distributed multi-agent systems use large language models to enable collaborative intelligence while preserving privacy, yet systematic evaluations of long-term memory under network constraints remain limited. This study presents a flexible testbed comparing mem0, a vector-based memory framework, and Graphiti, a graph-based knowledge graph, using the LOCOMO long-context benchmark. Experiments were conducted under unconstrained and constrained network conditions, measuring computational, financial, and accuracy metrics. Results indicate that mem0 significantly outperforms Graphiti in efficiency, with faster loading times, lower resource consumption, and minimal network overhead, while accuracy differences are not statistically significant. Applying a statistical pareto efficiency framework, mem0 is identified as the optimal choice that balances cost and accuracy in DMAS.
Paper Structure (14 sections, 4 figures, 7 tables)

This paper contains 14 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Theoretical framework and literature study
  3. Research questions and hypothesis
  4. Method
  5. Empirical approach
  6. Testbed architecture
  7. Results and analysis
  8. Discussion
  9. Economic analysis and total cost of ownership
  10. Formalizing the balance between cost and accuracy
  11. Comparative evaluation
  12. Limitations and future work
  13. Conclusion
  14. Appendix

Figures (4)

  • Figure 1: testbed system architecture
  • Figure 2: accuracy per experiment and memory
  • Figure 3: financial cost of computation and tokens consumed during the unconstrained experiment
  • Figure 4: financial cost of computation and tokens consumed during the constrained experiment