Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Addressing LLM Diversity by Infusing Random Concepts

Pulin Agrawal, Prasoon Goyal

TL;DR

The paper tackles the long-tail diversity problem in large language models (LLMs) by proposing a lightweight prompting technique that infuses random concepts into prompts. It introduces a systematic evaluation protocol using list-based prompts and measures diversity via $Count(p_i)=|\text{set}(L_{p_i})|$ and $H(L_p)$, analyzing effects across multiple models and prompt styles. Empirical results show that prepending random words or random sentences increases both the number of unique responses and the entropy of the response distribution, with statistical significance across ordered and unordered settings and regardless of temperature, indicating orthogonality to sampling settings. This approach is simple, composable with other methods, and offers a scalable direction for improving LLM diversity and for benchmarking diversity across domains.

Abstract

Large language models (LLMs) are known to produce outputs with limited diversity. In this work, we study whether infusing random concepts in the prompts can improve the diversity of the generated outputs. To benchmark the approach, we design a systematic evaluation protocol which involves prompting an LLM with questions of the form "Name 10 Hollywood actors", and analyzing diversity measures of the resulting LLM outputs. Our experiments on multiple LLMs show that prepending random words/sentences unrelated to the prompt result in greater diversity in the outputs of LLMs. We believe that this promising result and the evaluation protocol opens up interesting avenues for future work, such as how infusing randomness into LLMs could be applied to other domains. Further, the evaluation protocol could also inspire research into benchmarking LLM diversity more systematically.

Addressing LLM Diversity by Infusing Random Concepts

TL;DR

The paper tackles the long-tail diversity problem in large language models (LLMs) by proposing a lightweight prompting technique that infuses random concepts into prompts. It introduces a systematic evaluation protocol using list-based prompts and measures diversity via and , analyzing effects across multiple models and prompt styles. Empirical results show that prepending random words or random sentences increases both the number of unique responses and the entropy of the response distribution, with statistical significance across ordered and unordered settings and regardless of temperature, indicating orthogonality to sampling settings. This approach is simple, composable with other methods, and offers a scalable direction for improving LLM diversity and for benchmarking diversity across domains.

Abstract

Large language models (LLMs) are known to produce outputs with limited diversity. In this work, we study whether infusing random concepts in the prompts can improve the diversity of the generated outputs. To benchmark the approach, we design a systematic evaluation protocol which involves prompting an LLM with questions of the form "Name 10 Hollywood actors", and analyzing diversity measures of the resulting LLM outputs. Our experiments on multiple LLMs show that prepending random words/sentences unrelated to the prompt result in greater diversity in the outputs of LLMs. We believe that this promising result and the evaluation protocol opens up interesting avenues for future work, such as how infusing randomness into LLMs could be applied to other domains. Further, the evaluation protocol could also inspire research into benchmarking LLM diversity more systematically.
Paper Structure (11 sections, 5 equations, 4 figures, 3 tables)

This paper contains 11 sections, 5 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Infusing Random Concepts to Improve LLM Diversity
  3. Experiments and Results
  4. Discussion
  5. Limitations
  6. Appendix
  7. Ablation
  8. Data
  9. Artifact Terms of Use
  10. Model Size and Budget
  11. AI Use

Figures (4)

  • Figure 1: The long-tail problem of LLMs: Common responses have much higher frequency of generation by LLMs, while less common responses (on the long tail) are undersampled.
  • Figure 2: Histogram of unique items in the responses from all models with and without a random word context across the 'ordered' and 'unordered' settings.
  • Figure 3: Average frequency vs. frequency ranks for all models across the 'ordered' and 'unordered' settings.
  • Figure 4: Average frequency vs. frequency ranks for Gemma3 4b model in unordered setting for higher ranks from Figure \ref{['fig:log-freq-rank']}.