Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CogCanvas: Verbatim-Grounded Artifact Extraction for Long LLM Conversations

Tao An

TL;DR

CogCanvas tackles the fidelity gap in long-context LLM conversations by extracting verbatim-grounded cognitive artifacts and organizing them in a temporal-aware graph, offering a training-free baseline that avoids context compression. It uses a three-part pipeline: artifact extraction (with a two-pass gleaning step), graph construction with augmented edges, and adaptive injection via a hybrid semantic-keyword retrieval plus reranking strategy to generate reasoning-aware prompts. On LoCoMo and synthetic benchmarks, CogCanvas outperforms standard baselines, with especially large gains in temporal reasoning and solid performance on single-hop and multi-hop retrieval, while maintaining immediate deployability compared to stateful memory systems. The results highlight a practical trade-off: CogCanvas delivers strong, deployment-friendly memory capabilities and competitive accuracy, though heavily optimized stateful systems can still surpass it; hybrid strategies may yield the best of both worlds. The work provides a concrete, readily integrable approach for real-world long-context applications such as project planning, technical support, and collaborative coding, with public code and benchmarks available for reproducibility.

Abstract

Conversation summarization loses nuanced details: when asked about coding preferences after 40 turns, summarization recalls "use type hints" but drops the critical constraint "everywhere" (19.0% exact match vs. 93.0% for our approach). We present CogCanvas, a training-free framework inspired by how teams use whiteboards to anchor shared memory. Rather than compressing conversation history, CogCanvas extracts verbatim-grounded artifacts (decisions, facts, reminders) and retrieves them via temporal-aware graph. On the LoCoMo benchmark (all 10 conversations from the ACL 2024 release), CogCanvas achieves the highest overall accuracy among training-free methods (32.4%), outperforming RAG (24.6%) by +7.8pp, with decisive advantages on complex reasoning tasks: +20.6pp on temporal reasoning (32.7% vs. 12.1% RAG) and +1.1pp on multi-hop questions (41.7% vs. 40.6% RAG). CogCanvas also leads on single-hop retrieval (26.6% vs. 24.6% RAG). Ablation studies reveal that BGE reranking contributes +7.7pp, making it the largest contributor to CogCanvas's performance. While heavily-optimized approaches achieve higher absolute scores through dedicated training (EverMemOS: ~92%), our training-free approach provides practitioners with an immediately-deployable alternative that significantly outperforms standard baselines. Code and data: https://github.com/tao-hpu/cog-canvas

CogCanvas: Verbatim-Grounded Artifact Extraction for Long LLM Conversations

TL;DR

CogCanvas tackles the fidelity gap in long-context LLM conversations by extracting verbatim-grounded cognitive artifacts and organizing them in a temporal-aware graph, offering a training-free baseline that avoids context compression. It uses a three-part pipeline: artifact extraction (with a two-pass gleaning step), graph construction with augmented edges, and adaptive injection via a hybrid semantic-keyword retrieval plus reranking strategy to generate reasoning-aware prompts. On LoCoMo and synthetic benchmarks, CogCanvas outperforms standard baselines, with especially large gains in temporal reasoning and solid performance on single-hop and multi-hop retrieval, while maintaining immediate deployability compared to stateful memory systems. The results highlight a practical trade-off: CogCanvas delivers strong, deployment-friendly memory capabilities and competitive accuracy, though heavily optimized stateful systems can still surpass it; hybrid strategies may yield the best of both worlds. The work provides a concrete, readily integrable approach for real-world long-context applications such as project planning, technical support, and collaborative coding, with public code and benchmarks available for reproducibility.

Abstract

Conversation summarization loses nuanced details: when asked about coding preferences after 40 turns, summarization recalls "use type hints" but drops the critical constraint "everywhere" (19.0% exact match vs. 93.0% for our approach). We present CogCanvas, a training-free framework inspired by how teams use whiteboards to anchor shared memory. Rather than compressing conversation history, CogCanvas extracts verbatim-grounded artifacts (decisions, facts, reminders) and retrieves them via temporal-aware graph. On the LoCoMo benchmark (all 10 conversations from the ACL 2024 release), CogCanvas achieves the highest overall accuracy among training-free methods (32.4%), outperforming RAG (24.6%) by +7.8pp, with decisive advantages on complex reasoning tasks: +20.6pp on temporal reasoning (32.7% vs. 12.1% RAG) and +1.1pp on multi-hop questions (41.7% vs. 40.6% RAG). CogCanvas also leads on single-hop retrieval (26.6% vs. 24.6% RAG). Ablation studies reveal that BGE reranking contributes +7.7pp, making it the largest contributor to CogCanvas's performance. While heavily-optimized approaches achieve higher absolute scores through dedicated training (EverMemOS: ~92%), our training-free approach provides practitioners with an immediately-deployable alternative that significantly outperforms standard baselines. Code and data: https://github.com/tao-hpu/cog-canvas
Paper Structure (57 sections, 5 equations, 5 figures, 10 tables)

This paper contains 57 sections, 5 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. CanvasObject Structure
  5. Graph Construction
  6. Extraction via LLM with Gleaning.
  7. Linking via Vector Similarity.
  8. Adaptive Injection
  9. Hybrid Retrieval with Adaptive Top-k.
  10. Two-Stage Retrieval with Reranking.
  11. Experiments
  12. Experimental Setup
  13. Benchmark Design.
  14. Evaluation Metrics.
  15. Baselines.
  16. ...and 42 more sections

Figures (5)

  • Figure 1: CogCanvas system architecture. Each conversation turn flows through the LLM Extractor to produce typed cognitive artifacts, which are linked by vector similarity into a semantic graph. At query time, the Vector Retriever selects relevant artifacts for Context Injection into the prompt.
  • Figure 2: Information retention comparison across context management methods.
  • Figure 3: Multi-hop Causal Reasoning: CogCanvas achieves 81.0% pass rate, outperforming GraphRAG (+41.0pp) and RAG (+25.5pp).
  • Figure 4: LoCoMo performance breakdown by question type.
  • Figure 5: Knowledge graph extracted from Next.js RFC discussion. Node colors indicate artifact types: DECISION (red), TODO (blue), KEY_FACT (green), INSIGHT (purple), and REMINDER (orange). Edge types show semantic relationships.