Vulcan: Instance-Optimal Systems Heuristics Through LLM-Driven Search

TL;DR

Vulcan addresses the lack of universal system heuristics by reframing heuristic design as an instance-specific code-synthesis problem powered by LLM-driven search. It introduces a three-step pipeline that separates policy from mechanism via Value and Rank interfaces and uses offline evolutionary search to generate executable, policy-like scoring functions. Case studies in cache eviction and memory tiering show substantial gains over state-of-the-art baselines (up to $69\%$ and $7.9\%$, respectively) while keeping runtime overhead predictable and the synthesized code interpretable. The work advocates principled abstractions and automated specialization as a practical path for scalable, high-performance systems design.

Abstract

Resource-management tasks in modern operating and distributed systems continue to rely primarily on hand-designed heuristics for tasks such as scheduling, caching, or active queue management. Designing performant heuristics is an expensive, time-consuming process that we are forced to continuously go through due to the constant flux of hardware, workloads and environments. We propose a new alternative: synthesizing instance-optimal heuristics -- specialized for the exact workloads and hardware where they will be deployed -- using code-generating large language models (LLMs). To make this synthesis tractable, Vulcan separates policy and mechanism through LLM-friendly, task-agnostic interfaces. With these interfaces, users specify the inputs and objectives of their desired policy, while Vulcan searches for performant policies via evolutionary search over LLM-generated code. This interface is expressive enough to capture a wide range of system policies, yet sufficiently constrained to allow even small, inexpensive LLMs to generate correct and executable code. We use Vulcan to synthesize performant heuristics for cache eviction and memory tiering, and find that these heuristics outperform all human-designed state-of-the-art algorithms by upto 69% and 7.9% in performance for each of these tasks respectively.

Figures (11)

• Figure 1: Count of CloudPhysics traces where each heuristic performs best (highest object hit rate). Tiny, small, and large caches correspond to 0.1%, 1%, and 10% of the trace footprint (i.e., number of unique objects in a trace); we ignore object sizes, similar to how sieve and s3-fifo report their results. The column "Others" includes SR_LRU cacheus, CR_LFU cacheus, Sieve sieve, and GDSF gdsf all of which had at least one trace in which they were the best heuristic.
• Figure 2: Three-step user-facing pipeline for instantiating Vulcan. All of these are part of the problem specification phase (Phase I) shown in Figure \ref{['fig:system-overview']}.
• Figure 3: End-to-end overview of Vulcan. Each of the inputs defined in Phase I are used by subsequent phases.
• Figure 4: Division of labour in Vulcan: users are responsible for implementing mechanisms for data collection and action (in green); LLMs synthesize the scoring functions (in yellow); Vulcan uses templates to assemble these LLM-generated functions into an executable policy module (in blue).
• Figure 5: Automated instance generation and runtime policy selection with Vulcan.