SemanticALLI: Caching Reasoning, Not Just Responses, in Agentic Systems

TL;DR

SemanticALLI targets the Latency-Utility Gap in enterprise agentic pipelines by caching internal reasoning rather than final outputs. It introduces a two-stage decomposition, Analytic Intent Resolution (AIR) and Visualization Synthesis (VS), with a Hybrid Retrieval Engine that combines exact hashing, dense semantic indexing, and lexical constraints. Empirical results show VS achieves up to $83.10\%$ hit rate and AIR up to $38.7\%$, bypassing thousands of LLM calls and reducing latency, with tokens per invocation dramatically lowered for cached steps. This internal caching yields substantial token and latency savings compared with monolithic prompt->output caching, suggesting a generalizable principle for multi-step AI workflows in BI and beyond. The work points to future directions in broader deployment, cross-tenant reuse, and more sophisticated invalidation strategies.

Abstract

Agentic AI pipelines suffer from a hidden inefficiency: they frequently reconstruct identical intermediate logic, such as metric normalization or chart scaffolding, even when the user's natural language phrasing is entirely novel. Conventional boundary caching fails to capture this inefficiency because it treats inference as a monolithic black box. We introduce SemanticALLI, a pipeline-aware architecture within Alli (PMG's marketing intelligence platform), designed to operationalize redundant reasoning. By decomposing generation into Analytic Intent Resolution (AIR) and Visualization Synthesis (VS), SemanticALLI elevates structured intermediate representations (IRs) to first-class, cacheable artifacts. The impact of caching within the agentic loop is substantial. In our evaluation, baseline monolithic caching caps at a 38.7% hit rate due to linguistic variance. In contrast, our structured approach allows for an additional stage, the Visualization Synthesis stage, to achieve an 83.10% hit rate, bypassing 4,023 LLM calls with a median latency of just 2.66 ms. This internal reuse reduces total token consumption, offering a practical lesson for AI system design: even when users rarely repeat themselves, the pipeline often does, at stable, structured checkpoints where caching is most reliable.

Figures (3)

• Figure 1: Projected token usage with SemanticALLI caching vs. without caching (500 prompts, $\tau=0.90$). "Without caching" is a counterfactual in which all AIR prompts and VS invocations are LLM-backed at baseline per-call token costs (AIR: 6,414.55 tokens/prompt; VS: 5,524.78 tokens/invocation). "With caching" uses the observed SemanticALLI costs (AIR: 3,925.71 tokens/prompt; VS: 933.54 tokens/invocation). The projection uses the observed rate of VS invocations per user prompt ($4{,}841/500 \approx 9.68$), yielding average tokens per user prompt of 59,906 without caching vs. 12,964 with caching (78.4% reduction). Percent labels denote token reduction relative to the counterfactual.
• Figure 2: Projected API cost with VS caching vs. without caching per 10,000 calls. “With caching” assumes the measured call reduction at $\tau = 0.90$, retaining 21.04% of LLM-backed calls ($\approx$2,104 of 10,000). Costs are computed using per-token list pricing (input/output) and the observed mean token footprint per call (2,788 input tokens; 2,979 output tokens).
• Figure 3: Hit rate comparison for discussed caching architectures. Cache Saver is shown at the midpoint of its reported 21--60% range. Asteria reports workload-specific hit rates, shown as three points. SemanticALLI is marked with a star.