Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-42: Enabling Determinism in LLM Inference with Verified Speculation

Raja Gond, Aditya K Kamath, Arkaprava Basu, Ramachandran Ramjee, Ashish Panwar

TL;DR

This work addresses non-determinism in LLM inference caused by floating-point non-associativity and dynamic batching. It proposes LLM-42, a scheduling-based decode–verify–rollback approach that enforces determinism selectively by replaying a fixed window of tokens under a fixed reduction schedule, while keeping non-deterministic decoding as the fast path. The key contributions include a detailed DVR design, a grouped verification strategy to balance verification and recomputation costs, and techniques to keep the KV cache consistent. Evaluation on Llama-3.1-8B-Instruct shows LLM-42 delivers near-peak throughput for non-deterministic traffic with modest overhead when determinism is required, outperforming batch-invariant determinism-heavy baselines and enabling selective determinism for real-world deployments.

Abstract

In LLM inference, the same prompt may yield different outputs across different runs. At the system level, this non-determinism arises from floating-point non-associativity combined with dynamic batching and GPU kernels whose reduction orders vary with batch size. A straightforward way to eliminate non-determinism is to disable dynamic batching during inference, but doing so severely degrades throughput. Another approach is to make kernels batch-invariant; however, this tightly couples determinism to kernel design, requiring new implementations. This coupling also imposes fixed runtime overheads, regardless of how much of the workload actually requires determinism. Inspired by ideas from speculative decoding, we present LLM-42, a scheduling-based approach to enable determinism in LLM inference. Our key observation is that if a sequence is in a consistent state, the next emitted token is likely to be consistent even with dynamic batching. Moreover, most GPU kernels use shape-consistent reductions. Leveraging these insights, LLM-42 decodes tokens using a non-deterministic fast path and enforces determinism via a lightweight verify-rollback loop. The verifier replays candidate tokens under a fixed-shape reduction schedule, commits those that are guaranteed to be consistent across runs, and rolls back those violating determinism. LLM-42 mostly re-uses existing kernels unchanged and incurs overhead only in proportion to the traffic that requires determinism.

LLM-42: Enabling Determinism in LLM Inference with Verified Speculation

TL;DR

This work addresses non-determinism in LLM inference caused by floating-point non-associativity and dynamic batching. It proposes LLM-42, a scheduling-based decode–verify–rollback approach that enforces determinism selectively by replaying a fixed window of tokens under a fixed reduction schedule, while keeping non-deterministic decoding as the fast path. The key contributions include a detailed DVR design, a grouped verification strategy to balance verification and recomputation costs, and techniques to keep the KV cache consistent. Evaluation on Llama-3.1-8B-Instruct shows LLM-42 delivers near-peak throughput for non-deterministic traffic with modest overhead when determinism is required, outperforming batch-invariant determinism-heavy baselines and enabling selective determinism for real-world deployments.

Abstract

In LLM inference, the same prompt may yield different outputs across different runs. At the system level, this non-determinism arises from floating-point non-associativity combined with dynamic batching and GPU kernels whose reduction orders vary with batch size. A straightforward way to eliminate non-determinism is to disable dynamic batching during inference, but doing so severely degrades throughput. Another approach is to make kernels batch-invariant; however, this tightly couples determinism to kernel design, requiring new implementations. This coupling also imposes fixed runtime overheads, regardless of how much of the workload actually requires determinism. Inspired by ideas from speculative decoding, we present LLM-42, a scheduling-based approach to enable determinism in LLM inference. Our key observation is that if a sequence is in a consistent state, the next emitted token is likely to be consistent even with dynamic batching. Moreover, most GPU kernels use shape-consistent reductions. Leveraging these insights, LLM-42 decodes tokens using a non-deterministic fast path and enforces determinism via a lightweight verify-rollback loop. The verifier replays candidate tokens under a fixed-shape reduction schedule, commits those that are guaranteed to be consistent across runs, and rolls back those violating determinism. LLM-42 mostly re-uses existing kernels unchanged and incurs overhead only in proportion to the traffic that requires determinism.
Paper Structure (24 sections, 12 figures, 5 tables)

This paper contains 24 sections, 12 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Large Language Models
  4. Non-determinism in LLM Inference
  5. Defeating Non-determinism
  6. Observations
  7. LLM-42
  8. Overall Design
  9. Decode-verify-rollback (DVR)
  10. Case-1: When verification succeeds.
  11. Case-2: When verification fails.
  12. Making KV cache consistent.
  13. Guaranteed forward progress.
  14. Grouped Verification
  15. Discussion
  16. ...and 9 more sections

Figures (12)

  • Figure 1: Overview of LLM-42.
  • Figure 2: High-level architecture of LLMs.
  • Figure 3: GEMM kernels compute dot products using standard accumulation or split-K parallelization. While split-K increases parallelism, it alters the reduction tree based on K.
  • Figure 4: Performance comparison between batch-invariant vs. non-batch-invariant kernels.
  • Figure 5: Decode throughput under different scenarios.
  • ...and 7 more figures