Statistical Independence Aware Caching for LLM Workflows

TL;DR

The paper addresses the conflict between caching LLM outputs and maintaining statistical independence in probabilistic workflows. It introduces Mnimi, a cache design pattern that encodes independence constraints as LLM reference types (Repeatable and Independent) and uses decorators to compose them in modular workflows. A Python implementation demonstrates configurable backends and sample transformations, with a case study on SpecFix showing reduced cost and improved reproducibility. The work provides a practical framework for building statistically sound, cacheable LLM pipelines and outlines avenues for semantic caching and concurrency enhancements.

Abstract

Large language models (LLMs) inference is both expensive and slow. Local caching of responses offers a practical solution to reduce the cost and latency of LLM queries. In research contexts, caching also enhances reproducibility and provides flexibility for experimentation. However, naive reuse of cached responses compromises statistical independence, a critical property for probabilistic workflows. In applications of LLM for code, it underpins performance metrics such as Pass@k and uncertainty estimation, as well as algorithms like program repair loops and retries. Existing LLM caching systems lack ways to enforce statistical independence constraints. To address this, we introduce Mnimi, a cache design pattern that supports modular LLM workflows while ensuring statistical integrity at the component level. Its core innovation lies in encapsulating statistical constraints within the type of LLM references, allowing users to manage and transform these types according to the scope and requirements of their algorithm. We implemented this design pattern in Python using a combination of decorators and iterators over infinite sequences. A case study on SpecFix, an recent automated program specification repair system, highlights how Mnimi improves reproducibility, ease of debugging, time and cost efficiency while preserving statistical correctness.

Figures (2)

• Figure 1: An example showing the different behaviors of Independent vs. Repeatable. In the "Iterator State Transition" column, the first sub-row of it shows the sequence state returned by calling sample and the second sub-row shows the state after iterating to retrieve elements, while "→" denotes the pointer to the next element inside the iterator.
• Figure 2: Structure of Mnimi design pattern that combines decorators and iterators over infinite sequences of responses to encode statistical independence constraints in a modular fashion. Mnimi provides two concrete implementations of Repeatable: an in-memory cache InMemory and an on-disk cache Persistent.