Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unveiling and Manipulating Prompt Influence in Large Language Models

Zijian Feng, Hanzhang Zhou, Zixiao Zhu, Junlang Qian, Kezhi Mao

TL;DR

Token Distribution Dynamics is applied to two prompt manipulation tasks for controlled text generation: zero-shot toxic language suppression and sentiment steering, demonstrating TDD's proficiency in identifying both toxic and sentimental cues in prompts, subsequently mitigating toxicity or modulating sentiment in the generated content.

Abstract

Prompts play a crucial role in guiding the responses of Large Language Models (LLMs). However, the intricate role of individual tokens in prompts, known as input saliency, in shaping the responses remains largely underexplored. Existing saliency methods either misalign with LLM generation objectives or rely heavily on linearity assumptions, leading to potential inaccuracies. To address this, we propose Token Distribution Dynamics (TDD), a \textcolor{black}{simple yet effective} approach to unveil and manipulate the role of prompts in generating LLM outputs. TDD leverages the robust interpreting capabilities of the language model head (LM head) to assess input saliency. It projects input tokens into the embedding space and then estimates their significance based on distribution dynamics over the vocabulary. We introduce three TDD variants: forward, backward, and bidirectional, each offering unique insights into token relevance. Extensive experiments reveal that the TDD surpasses state-of-the-art baselines with a big margin in elucidating the causal relationships between prompts and LLM outputs. Beyond mere interpretation, we apply TDD to two prompt manipulation tasks for controlled text generation: zero-shot toxic language suppression and sentiment steering. Empirical results underscore TDD's proficiency in identifying both toxic and sentimental cues in prompts, subsequently mitigating toxicity or modulating sentiment in the generated content.

Unveiling and Manipulating Prompt Influence in Large Language Models

TL;DR

Token Distribution Dynamics is applied to two prompt manipulation tasks for controlled text generation: zero-shot toxic language suppression and sentiment steering, demonstrating TDD's proficiency in identifying both toxic and sentimental cues in prompts, subsequently mitigating toxicity or modulating sentiment in the generated content.

Abstract

Prompts play a crucial role in guiding the responses of Large Language Models (LLMs). However, the intricate role of individual tokens in prompts, known as input saliency, in shaping the responses remains largely underexplored. Existing saliency methods either misalign with LLM generation objectives or rely heavily on linearity assumptions, leading to potential inaccuracies. To address this, we propose Token Distribution Dynamics (TDD), a \textcolor{black}{simple yet effective} approach to unveil and manipulate the role of prompts in generating LLM outputs. TDD leverages the robust interpreting capabilities of the language model head (LM head) to assess input saliency. It projects input tokens into the embedding space and then estimates their significance based on distribution dynamics over the vocabulary. We introduce three TDD variants: forward, backward, and bidirectional, each offering unique insights into token relevance. Extensive experiments reveal that the TDD surpasses state-of-the-art baselines with a big margin in elucidating the causal relationships between prompts and LLM outputs. Beyond mere interpretation, we apply TDD to two prompt manipulation tasks for controlled text generation: zero-shot toxic language suppression and sentiment steering. Empirical results underscore TDD's proficiency in identifying both toxic and sentimental cues in prompts, subsequently mitigating toxicity or modulating sentiment in the generated content.
Paper Structure (49 sections, 7 equations, 6 figures, 17 tables)

This paper contains 49 sections, 7 equations, 6 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Input saliency for language models
  4. Contrastive explanations for LLMs
  5. Unveil Prompt Influence in Large Language Models through Token Distribution Dynamics
  6. Problem Statement
  7. Token Distribution: A New Lens for Analyzing LLMs
  8. Forward token distribution as input saliency
  9. Backward token distribution as input saliency
  10. Bidirectional token distribution as input saliency
  11. Extensions of TDD
  12. Experiments
  13. Dataset
  14. Evaluation Metrics
  15. Choice of LLMs
  16. ...and 34 more sections

Figures (6)

  • Figure 1: Framework of TDD. It first employs the LM head to project token representations into the embedding space and then evaluates the significance of tokens through three distinctive variants from various directions. This illustration elucidates that the LLM's generation of '' guys" instead of '' guy" is primarily attributed to the presence of the word '' those" in the prompt.
  • Figure 2: Qualitative and quantitative results to study the token distributions for various LLMs.
  • Figure 3: Visualization of different methods to explain why the LLM generates '' were" instead of '' was". A deeper shade of red indicates a higher weight.
  • Figure 4: Visualization of different methods to explain why the LLM generates '' waiters" instead of '' jackets". A deeper shade of red indicates a higher weight.
  • Figure 5: Visualization of toxic language suppression. We feed the prompt '' Another caption says '' We gonna f—the" to GPT2 and then observe the outputs of the SOTA FFNControl and our TDD.
  • ...and 1 more figures