Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference

TL;DR

The paper tackles the high cost of deploying frontier LLMs for complex reasoning by enabling cost-efficient collaboration with smaller, cheaper SLMs. It introduces LLM Shepherding, where the LLM provides a short prefix (a hint) that the SLM uses to generate the final answer, enabling a continuous cost-accuracy trade-off rather than binary routing or cascading. A two-stage predictor learns when a hint is needed and how many tokens to request, supporting both proactive and reactive (cascading) modes. Across GSM8K, CNK12, HumanEval, and MBPP, shepherding achieves substantial cost reductions (up to 94% vs LLM-only) and outperforms routing and cascading baselines in ACE, with strong generalization to unseen domains and minimal latency overhead in practical settings.

Abstract

Large Language Models (LLMs) deliver state-of-the-art performance on complex reasoning tasks, but their inference costs limit deployment at scale. Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy. Existing approaches - routing and cascading - treat the LLM as an all-or-nothing resource: either the query bypasses the LLM entirely, or the LLM generates a complete response at full cost. We introduce LLM Shepherding, a framework that requests only a short prefix (a hint) from the LLM and provides it to SLM. This simple mechanism is surprisingly effective for math and coding tasks: even hints comprising 10-30% of the full LLM response improve SLM accuracy significantly. Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making. We develop a two-stage predictor that jointly determines whether a hint is needed and how many tokens to request. On the widely-used mathematical reasoning (GSM8K, CNK12) and code generation (HumanEval, MBPP) benchmarks, Shepherding reduces costs by 42-94% relative to LLM-only inference. Compared to state-of-the-art routing and cascading baselines, shepherding delivers up to 2.8x cost reduction while matching accuracy. To our knowledge, this is the first work to exploit token-level budget control for SLM-LLM collaboration.

Figures (7)

• Figure 1: Two modes of LLM Shepherding. Both modes request only the first $n$ tokens from the LLM as a hint, which the SLM uses to output the response. (a) Proactive: hint decision made upfront. (b) Reactive: hint requested only after SLM failure.
• Figure 2: SLM (Llama-3.2-3B-Instruct) accuracy as a function of LLM (Llama-3.3-70B-Versatile) hint size. Even small hints (10--30% of the full LLM response) yield substantial accuracy gains, with diminishing returns beyond 60%. This motivates shepherding: requesting hints rather than full LLM responses. Dashed lines indicate the LLM accuracy for each dataset.
• Figure 3: Minimum cost to achieve 90% of LLM accuracy on each dataset. Each method is tuned to its optimal operating point, which meets or exceeds the accuracy requirement (shown above bars).
• Figure 4: Outlier distribution for CNK12 dataset.
• Figure 5: Outlier distribution for GSM8K dataset.

Theorems & Definitions (3)

• Proposition 2.1: Oracle Costs
• Corollary 2.2
• Remark 2.3