Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access

TL;DR

The paper tackles enforcing task-specific constraints when the strongest LLMs are black boxes without access to logits. It proposes SketchGCD, a two-phase approach that sketches a constraint-violating output with a strong blackbox LLM and then refines it with a smaller, locally hosted constrained decoder to satisfy the constraints. Across closed information extraction and constituency parsing, SketchGCD improves constraint satisfaction and task metrics relative to unconstrained prompting, and remains competitive with logit-based constrained decoding when logits are available. The method is modular, complementary to existing constrained decoding techniques, and enables robust constrained generation on systems with limited model transparency, albeit with added computational overhead and limitations to structural constraints.

Abstract

Constrained decoding, a technique for enforcing constraints on language model outputs, offers a way to control text generation without retraining or architectural modifications. Its application is, however, typically restricted to models that give users access to next-token distributions (usually via softmax logits), which poses a limitation with blackbox large language models (LLMs). This paper introduces sketch-guided constrained decoding (SGCD), a novel approach to constrained decoding for blackbox LLMs, which operates without access to the logits of the blackbox LLM. SGCD utilizes a locally hosted auxiliary model to refine the output of an unconstrained blackbox LLM, effectively treating this initial output as a "sketch" for further elaboration. This approach is complementary to traditional logit-based techniques and enables the application of constrained decoding in settings where full model transparency is unavailable. We demonstrate the efficacy of SGCD through experiments in closed information extraction and constituency parsing, showing how it enhances the utility and flexibility of blackbox LLMs for complex NLP tasks.

Figures (6)

• Figure 1: Overview of sketch-guided constrained decoding (SketchGCD). In the initial sketching phase, a blackbox LLM generates a preliminary "sketch" answer without applying any constraints. Then, in the constrained decoding phase, an auxiliary model, the constrained decoder, refines the sketch. The refined, final output respects the specified constraints by construction.
• Figure 2: Instructions for parsing tasks.
• Figure 3: Relation distribution in WikiNRE before and after stratification.
• Figure 4: Instructions for parsing tasks.
• Figure 5: Parse trees for the sentence "I saw a fox".