Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning

Long Mai, Julie Carson-Berndsen

TL;DR

The paper tackles the problem of declining linguistic diversity and representation bias in Large Language Models. It introduces Possibility Exploration Fine-Tuning (PEFT), a task-agnostic, post-hoc fine-tuning method that trains models to output multiple diverse responses to a single prompt by leveraging one-to-many and possibility-indexed datasets, without increasing inference latency. PEFT combines likelihood and unlikelihood objectives to encourage semantic diversity while preserving coherence, and it supports controlled inference by sampling possibility numbers. Empirical results on open-domain dialogue and story generation show that PEFT yields a superior diversity-coherence-latency balance and reduces demographic bias, with evidence from quantified metrics and bias-related tests; a GitHub repository provides implementations and data. The work suggests a practical path toward more inclusive and varied LLM outputs, while noting the requirement of task-specific fine-tuning data and offering directions for integrating PEFT into broader instruction-tuning frameworks.

Abstract

While Large Language Models (LLMs) have made significant strides in replicating human-like abilities, there are concerns about a reduction in the linguistic diversity of their outputs. This results in the homogenization of viewpoints and perspectives, as well as the underrepresentation of specific demographic groups. Although several fine-tuning and prompting techniques have been suggested to tackle the issue, they are often tailored to specific tasks or come with a substantial increase in computational cost and latency. This makes them challenging to apply to applications that demand very low latency, such as chatbots and virtual assistants. We propose Possibility Exploration Fine-Tuning (PEFT), a task-agnostic framework that enhances the text diversity of LLMs without increasing latency or computational cost. Given the same prompt, models fine-tuned with PEFT can simultaneously generate multiple diverse responses, each corresponding with a controllable possibility number. Experiments on dialogue and story generation tasks demonstrate that PEFT significantly enhances the diversity of LLM outputs, as evidenced by lower similarity between candidate responses. Since PEFT emphasizes semantic diversity over lexical diversity, it can also notably reduce demographic bias in dialogue systems. The implementations and datasets are available in our repository: https://github.com/mailong25/peft_diversity

Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning

TL;DR

The paper tackles the problem of declining linguistic diversity and representation bias in Large Language Models. It introduces Possibility Exploration Fine-Tuning (PEFT), a task-agnostic, post-hoc fine-tuning method that trains models to output multiple diverse responses to a single prompt by leveraging one-to-many and possibility-indexed datasets, without increasing inference latency. PEFT combines likelihood and unlikelihood objectives to encourage semantic diversity while preserving coherence, and it supports controlled inference by sampling possibility numbers. Empirical results on open-domain dialogue and story generation show that PEFT yields a superior diversity-coherence-latency balance and reduces demographic bias, with evidence from quantified metrics and bias-related tests; a GitHub repository provides implementations and data. The work suggests a practical path toward more inclusive and varied LLM outputs, while noting the requirement of task-specific fine-tuning data and offering directions for integrating PEFT into broader instruction-tuning frameworks.

Abstract

While Large Language Models (LLMs) have made significant strides in replicating human-like abilities, there are concerns about a reduction in the linguistic diversity of their outputs. This results in the homogenization of viewpoints and perspectives, as well as the underrepresentation of specific demographic groups. Although several fine-tuning and prompting techniques have been suggested to tackle the issue, they are often tailored to specific tasks or come with a substantial increase in computational cost and latency. This makes them challenging to apply to applications that demand very low latency, such as chatbots and virtual assistants. We propose Possibility Exploration Fine-Tuning (PEFT), a task-agnostic framework that enhances the text diversity of LLMs without increasing latency or computational cost. Given the same prompt, models fine-tuned with PEFT can simultaneously generate multiple diverse responses, each corresponding with a controllable possibility number. Experiments on dialogue and story generation tasks demonstrate that PEFT significantly enhances the diversity of LLM outputs, as evidenced by lower similarity between candidate responses. Since PEFT emphasizes semantic diversity over lexical diversity, it can also notably reduce demographic bias in dialogue systems. The implementations and datasets are available in our repository: https://github.com/mailong25/peft_diversity

Paper Structure

This paper contains 30 sections, 5 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Baselines
  4. Problem definition
  5. Decoding methods
  6. Prompting methods
  7. Fine-tuning methods
  8. Proposed method
  9. One-to-many dataset
  10. Possibility exploration dataset
  11. PEFT
  12. Experiments
  13. Tasks
  14. Metrics
  15. Diversity
  16. ...and 15 more sections

Figures (3)

  • Figure 1: An example of List Prompting for open-domain dialogue generation
  • Figure 2: An simplified example of a PE training batch with added possibility numbers. Full template can be found in Appendix \ref{['app:peft']}.
  • Figure 3: Persona demographic distributions extracted from 300 conversations with Mistral base and its fine-tuned models. All models use temperature sampling with $t=1.0$.