Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval

Jesper Derehag, Carlos Calva, Timmy Ghiurau

Abstract

Recent conversational memory systems invest heavily in LLM-based structuring at ingestion time and learned retrieval policies at query time. We show that neither is necessary. SmartSearch retrieves from raw, unstructured conversation history using a fully deterministic pipeline: NER-weighted substring matching for recall, rule-based entity discovery for multi-hop expansion, and a CrossEncoder+ColBERT rank fusion stage -- the only learned component -- running on CPU in ~650ms. Oracle analysis on two benchmarks identifies a compilation bottleneck: retrieval recall reaches 98.6%, but without intelligent ranking only 22.5% of gold evidence survives truncation to the token budget. With score-adaptive truncation and no per-dataset tuning, SmartSearch achieves 93.5% on LoCoMo and 88.4% on LongMemEval-S, exceeding all known memory systems under the same evaluation protocol on both benchmarks while using 8.5x fewer tokens than full-context baselines.

SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval

Abstract

Recent conversational memory systems invest heavily in LLM-based structuring at ingestion time and learned retrieval policies at query time. We show that neither is necessary. SmartSearch retrieves from raw, unstructured conversation history using a fully deterministic pipeline: NER-weighted substring matching for recall, rule-based entity discovery for multi-hop expansion, and a CrossEncoder+ColBERT rank fusion stage -- the only learned component -- running on CPU in ~650ms. Oracle analysis on two benchmarks identifies a compilation bottleneck: retrieval recall reaches 98.6%, but without intelligent ranking only 22.5% of gold evidence survives truncation to the token budget. With score-adaptive truncation and no per-dataset tuning, SmartSearch achieves 93.5% on LoCoMo and 88.4% on LongMemEval-S, exceeding all known memory systems under the same evaluation protocol on both benchmarks while using 8.5x fewer tokens than full-context baselines.
Paper Structure (40 sections, 1 figure, 10 tables)

This paper contains 40 sections, 1 figure, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Long-Context Memory Systems
  4. Retrieval Methods
  5. Multi-Hop Retrieval and Question Answering
  6. Retrieval-Augmented Generation
  7. Reranking
  8. Method
  9. Evaluation
  10. Oracle Trace Derivation
  11. Parse and Retrieve
  12. Query Expansion
  13. Rank
  14. Truncate
  15. Experimental Setup
  16. ...and 25 more sections

Figures (1)

  • Figure 1: SmartSearch pipeline. Stages 1--2 are fully deterministic; Stage 3 (Rank) is the only learned component, running two models in parallel on CPU. Dashed arrow: optional multi-hop entity expansion (3% of queries).