Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Forcing Diffuse Distributions out of Language Models

Yiming Zhang, Avi Schwarzschild, Nicholas Carlini, Zico Kolter, Daphne Ippolito

TL;DR

The paper addresses the problem that instruction-tuned language models produce low-entropy, uneven outputs when randomness or diversity is required. It introduces a distribution-matching fine-tuning framework, implemented with parameter-efficient LoRA, to diffuse model outputs over valid candidates. The method yields strong, transferable improvements in output diversity across simple tasks and complex synthetic data generation, without compromising core capabilities. This approach enables automated, high-diversity synthetic dataset creation and shows promise for debiasing and expanding open-ended generation, though it relies on structured output spaces and prefix-free target sets. Overall, the work demonstrates a practical route to making large language models more useful for tasks requiring diverse and representative outputs.

Abstract

Despite being trained specifically to follow user instructions, today's instructiontuned language models perform poorly when instructed to produce random outputs. For example, when prompted to pick a number uniformly between one and ten Llama-2-13B-chat disproportionately favors the number five, and when tasked with picking a first name at random, Mistral-7B-Instruct chooses Avery 40 times more often than we would expect based on the U.S. population. When these language models are used for real-world tasks where diversity of outputs is crucial, such as language model assisted dataset construction, their inability to produce diffuse distributions over valid choices is a major hurdle. In this work, we propose a fine-tuning method that encourages language models to output distributions that are diffuse over valid outcomes. The methods we introduce generalize across a variety of tasks and distributions and make large language models practical for synthetic dataset generation with little human intervention.

Forcing Diffuse Distributions out of Language Models

TL;DR

The paper addresses the problem that instruction-tuned language models produce low-entropy, uneven outputs when randomness or diversity is required. It introduces a distribution-matching fine-tuning framework, implemented with parameter-efficient LoRA, to diffuse model outputs over valid candidates. The method yields strong, transferable improvements in output diversity across simple tasks and complex synthetic data generation, without compromising core capabilities. This approach enables automated, high-diversity synthetic dataset creation and shows promise for debiasing and expanding open-ended generation, though it relies on structured output spaces and prefix-free target sets. Overall, the work demonstrates a practical route to making large language models more useful for tasks requiring diverse and representative outputs.

Abstract

Despite being trained specifically to follow user instructions, today's instructiontuned language models perform poorly when instructed to produce random outputs. For example, when prompted to pick a number uniformly between one and ten Llama-2-13B-chat disproportionately favors the number five, and when tasked with picking a first name at random, Mistral-7B-Instruct chooses Avery 40 times more often than we would expect based on the U.S. population. When these language models are used for real-world tasks where diversity of outputs is crucial, such as language model assisted dataset construction, their inability to produce diffuse distributions over valid choices is a major hurdle. In this work, we propose a fine-tuning method that encourages language models to output distributions that are diffuse over valid outcomes. The methods we introduce generalize across a variety of tasks and distributions and make large language models practical for synthetic dataset generation with little human intervention.
Paper Structure (27 sections, 6 equations, 6 figures, 8 tables)

This paper contains 27 sections, 6 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem setting
  4. Measuring output distributions
  5. Forcing Diffuse Probabilities
  6. Distribution matching as language model fine-tuning
  7. Parameter-efficient Fine-tuning
  8. Results on Simple Tasks
  9. Language models fail to follow instructions for randomness
  10. Optimizing for BabyNames and RNG diversifies generation
  11. Generalization Across Tasks
  12. Constructing More Diverse Synthetic Datasets
  13. Out-of-distribution training improves synthetic dataset diversity
  14. Diversifying distributions of categorical attributes
  15. Related Work
  16. ...and 12 more sections

Figures (6)

  • Figure 1: Language models do not produce diffuse probabilities. The output distributions of untuned Gemma, Llama-2, and Mistral deviate from what we would expect from natural/random distributions. Our tuning method addresses this by diffusing the output distribution over valid candidates. The horizontal axes above are sorted in descending order by probability of the output.
  • Figure 2: Models tuned on Random Number Generation demonstrate generalization to variations in both prompt format and number ranges. 95% confidence intervals are shown in plots.
  • Figure 3: Entropy in leave-one-out generalization. The title of each plot indicates which set of tasks we compute entropy over.
  • Figure 4: Fine-tuned Llama-2 model improves the diversity of synthetic biographies. We report coverage for categorical attributes in \ref{['fig:synth-bio-coverage']} and normalized unigram diversity of generated achievements and the entire biography in \ref{['fig:synth-bio-unigrams']}.
  • Figure 5: Fine-tuned Llama-2 models generate more diverse birth places in biographies. Each circle represents the birth place corresponding to a single biography in our dataset.
  • ...and 1 more figures