Inferring from Logits: Exploring Best Practices for Decoding-Free Generative Candidate Selection

TL;DR

This work formalizes decoding-free generative candidate selection, where task-level answers are inferred directly from the initial logits rather than through full autoregressive decoding. It provides a formal problem definition and a comprehensive evaluation across diverse LM architectures on tasks with both small and enormous candidate pools. Key findings show that the first decoding step logits are highly informative, that estimation methods can outperform full decoding on non-instruction-tuned models and achieve substantial speedups (roughly 25–57x), and that the choice of LM and task critically shapes performance. The results offer practical guidance for designing efficient, outcome-focused candidate selection in generative LMs and highlight tradeoffs between accuracy and throughput in real-world settings.

Abstract

Generative Language Models rely on autoregressive decoding to produce the output sequence token by token. Many tasks such as preference optimization, require the model to produce task-level output consisting of multiple tokens directly by selecting candidates from a pool as predictions. Determining a task-level prediction from candidates using the ordinary token-level decoding mechanism is constrained by time-consuming decoding and interrupted gradients by discrete token selection. Existing works have been using decoding-free candidate selection methods to obtain candidate probability from initial output logits over vocabulary. Though these estimation methods are widely used, they are not systematically evaluated, especially on end tasks. We introduce an evaluation of a comprehensive collection of decoding-free candidate selection approaches on a comprehensive set of tasks, including five multiple-choice QA tasks with a small candidate pool and four clinical decision tasks with a massive amount of candidates, some with 10k+ options. We evaluate the estimation methods paired with a wide spectrum of foundation LMs covering different architectures, sizes and training paradigms. The results and insights from our analysis inform the future model design.

Figures (9)

• Figure 1: Decoding-free generative candidate selection produces results without training specialized parameters, building index, or token-by-token decoding.
• Figure 2: Demonstration of generative candidate selection methods. (a) Using full decoding for tasks with answer candidates by generating output sequences token-by-token. The task is to identify diagnoses given the patient's medical record. (b) Candidate space, consisting of coded diagnoses. (c) Using decoding-free generative candidate selection method for the diagnoses task based on prior-decoding logits over vocabulary.
• Figure 3: Recall for the diagnosis task of various estimation methods using LLaMA3 8B Instruct while (a) using logits obtained from different output steps and (b) using logits calculated over different numbers of essential tokens selected from candidate sequences. The bars in (a) indicate the unique tokens for full decoded sequences at corresponding output steps, reflecting the diversity of the decoded tokens.
• Figure 4: Accuracy with respect to the average candidate length for two types of datasets using LLaMA3 Instruct 8B: 1(a) Accuracy concerning model size for the ARC dataset and Flan-T5 family on a log-log scale. 1(b) Model size ablation for GPT-2 family. 2(a) MMLU dataset with a limited number of candidates, and 2(b) clinical decision datasets with a massive number of candidates.
• Figure 5: Distribution of average candidate lengths for MCQA benchmarks.

Theorems & Definitions (1)

• Example 1