Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models

Camille Couturier, Spyros Mastorakis, Haiying Shen, Saravan Rajmohan, Victor Rühle

TL;DR

The paper tackles the high computational and bandwidth costs of real-time, multi-step QA with large language models by introducing semantic caching of intermediate contextual summaries. It presents a cache design that stores compressed, context-aware summaries, uses adaptive cosine similarity thresholds to determine hits, and supports both query-aware and query-agnostic summarization within retrieval-augmented generation workflows. Through comparisons with prompt-based and other caching methods, the study demonstrates that semantic caching can cut redundant computation by up to 50-60% while preserving answer quality on datasets such as NaturalQuestions, TriviaQA, and a synthetic ArXiv corpus. The work analyzes utility, latency, and cache-efficiency trade-offs, discusses deployment considerations, and identifies future directions for adaptive thresholds, scalability, privacy, and broader applications in real-world LLM systems.

Abstract

Large Language Models (LLMs) are increasingly deployed across edge and cloud platforms for real-time question-answering and retrieval-augmented generation. However, processing lengthy contexts in distributed systems incurs high computational overhead, memory usage, and network bandwidth. This paper introduces a novel semantic caching approach for storing and reusing intermediate contextual summaries, enabling efficient information reuse across similar queries in LLM-based QA workflows. Our method reduces redundant computations by up to 50-60% while maintaining answer accuracy comparable to full document processing, as demonstrated on NaturalQuestions, TriviaQA, and a synthetic ArXiv dataset. This approach balances computational cost and response quality, critical for real-time AI assistants.

Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models

TL;DR

The paper tackles the high computational and bandwidth costs of real-time, multi-step QA with large language models by introducing semantic caching of intermediate contextual summaries. It presents a cache design that stores compressed, context-aware summaries, uses adaptive cosine similarity thresholds to determine hits, and supports both query-aware and query-agnostic summarization within retrieval-augmented generation workflows. Through comparisons with prompt-based and other caching methods, the study demonstrates that semantic caching can cut redundant computation by up to 50-60% while preserving answer quality on datasets such as NaturalQuestions, TriviaQA, and a synthetic ArXiv corpus. The work analyzes utility, latency, and cache-efficiency trade-offs, discusses deployment considerations, and identifies future directions for adaptive thresholds, scalability, privacy, and broader applications in real-world LLM systems.

Abstract

Large Language Models (LLMs) are increasingly deployed across edge and cloud platforms for real-time question-answering and retrieval-augmented generation. However, processing lengthy contexts in distributed systems incurs high computational overhead, memory usage, and network bandwidth. This paper introduces a novel semantic caching approach for storing and reusing intermediate contextual summaries, enabling efficient information reuse across similar queries in LLM-based QA workflows. Our method reduces redundant computations by up to 50-60% while maintaining answer accuracy comparable to full document processing, as demonstrated on NaturalQuestions, TriviaQA, and a synthetic ArXiv dataset. This approach balances computational cost and response quality, critical for real-time AI assistants.
Paper Structure (28 sections, 7 figures, 3 tables)

This paper contains 28 sections, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt caching (KV-based approaches)
  4. Semantic Caching
  5. Other caching approaches
  6. Our approach vs. existing methods
  7. System design and Methodology
  8. Insights and System design
  9. Our Semantic Caching methodology
  10. Experimental setup
  11. Simulation design
  12. Datasets
  13. Similarity between questions
  14. Summaries
  15. Evaluation metrics
  16. ...and 13 more sections

Figures (7)

  • Figure 1: Simplified diagram of a chat-based assistant system with document retrieval: a user asks a question (1), the system retrieves relevant documents (2), summarizes them (3), and generates an answer (4). Summaries can be either query-agnostic (non-contextual) or query-aware (contextual). Such systems often involve multiple steps where intermediate outputs feed subsequent steps.
  • Figure 2: Instructions used to prompt the LLM to generate a general, non-contextual summary of a document
  • Figure 5: NaturalQuestions dataset: cosine similarity between pairs of questions asked about a document.
  • Figure 6: Ratios (compared to the document length) and lengths (in terms of words) of different summaries (10%, 25%, 50%, 100 words, 200 words, 400 words) obtained through GPT-4o-mini on TriviaQA documents.
  • Figure 7: Comparative analysis of retrieval methods across TriviaQA, NaturalQuestions, and synthetic ArXiv datasets, for a similarity threshold of 0.8 and summary length of 200 words. Each row represents a different metric as the number of questions increases: utility, cache hit rate, input/output tokens, and latency. Note: For the ArXiv dataset, full document answers serve as the reference, creating artificially high utility values for this method (shown as dotted lines).
  • ...and 2 more figures