Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Customizing Language Model Responses with Contrastive In-Context Learning

Xiang Gao, Kamalika Das

TL;DR

This work introduces contrastive in-context learning for language models, where positive exemplars of desired outputs and negative exemplars of undesired traits are provided to guide generation, accompanied by a reasoning step that analyzes the contrasts. The method combines two components—obtaining paired contrastive examples and forming prompts (including reasoning)—and is evaluated across real-world (StackExchange, Reddit) and synthetic datasets, showing consistent improvements over standard few-shot prompting. Key findings include that generated negatives can match human-written negatives in effectiveness, that combining contrastive examples with an explicit analysis yields the best performance, and that the approach improves prompt efficiency. The results suggest this framework offers a practical path to better align LLM outputs with user preferences in diverse content-generation tasks.

Abstract

Large language models (LLMs) are becoming increasingly important for machine learning applications. However, it can be challenging to align LLMs with our intent, particularly when we want to generate content that is preferable over others or when we want the LLM to respond in a certain style or tone that is hard to describe. To address this challenge, we propose an approach that uses contrastive examples to better describe our intent. This involves providing positive examples that illustrate the true intent, along with negative examples that show what characteristics we want LLMs to avoid. The negative examples can be retrieved from labeled data, written by a human, or generated by the LLM itself. Before generating an answer, we ask the model to analyze the examples to teach itself what to avoid. This reasoning step provides the model with the appropriate articulation of the user's need and guides it towards generting a better answer. We tested our approach on both synthesized and real-world datasets, including StackExchange and Reddit, and found that it significantly improves performance compared to standard few-shot prompting

Customizing Language Model Responses with Contrastive In-Context Learning

TL;DR

This work introduces contrastive in-context learning for language models, where positive exemplars of desired outputs and negative exemplars of undesired traits are provided to guide generation, accompanied by a reasoning step that analyzes the contrasts. The method combines two components—obtaining paired contrastive examples and forming prompts (including reasoning)—and is evaluated across real-world (StackExchange, Reddit) and synthetic datasets, showing consistent improvements over standard few-shot prompting. Key findings include that generated negatives can match human-written negatives in effectiveness, that combining contrastive examples with an explicit analysis yields the best performance, and that the approach improves prompt efficiency. The results suggest this framework offers a practical path to better align LLM outputs with user preferences in diverse content-generation tasks.

Abstract

Large language models (LLMs) are becoming increasingly important for machine learning applications. However, it can be challenging to align LLMs with our intent, particularly when we want to generate content that is preferable over others or when we want the LLM to respond in a certain style or tone that is hard to describe. To address this challenge, we propose an approach that uses contrastive examples to better describe our intent. This involves providing positive examples that illustrate the true intent, along with negative examples that show what characteristics we want LLMs to avoid. The negative examples can be retrieved from labeled data, written by a human, or generated by the LLM itself. Before generating an answer, we ask the model to analyze the examples to teach itself what to avoid. This reasoning step provides the model with the appropriate articulation of the user's need and guides it towards generting a better answer. We tested our approach on both synthesized and real-world datasets, including StackExchange and Reddit, and found that it significantly improves performance compared to standard few-shot prompting
Paper Structure (33 sections, 3 figures, 2 tables)

This paper contains 33 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contrastive In-Context Learning
  4. Obtaining Paired Contrastive Examples
  5. Using Labeled Feedback
  6. Using LLM-Generated Responses
  7. Using Automated Evaluator
  8. Forming the Prompt
  9. Contrastive Examples as Few-Shot Examples
  10. Reasoning and Analysis
  11. Experiments
  12. Datasets
  13. Human Preference Datasets
  14. StackExchange
  15. Reddit
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Contrastive examples provide positive examples that illustrate the true intent, along with negative examples that show what characteristics we want LLMs to avoid.
  • Figure 2: Contrastive in-context learning prompting utilize contrastive few-shot examples, which consists of a "postitive" example and a "negative" example for each demonstration input, and a prompt to elicit analysis of the characteristics of the positive/negative examples, before generating an output for the new input. This strategy can be applied to both conversational and non-conversational LLMs. For non-conversational LLMs, the labels and instructions are provided as system messages (the gear symbols).
  • Figure 3: The dependence of the performance on prompt length. For standard few-shot, we experimented with $k=1,2,3,4$, shown by the four black dots from left to right in each subfigure. For contrastive prompts, empty circles denotes prompts using generated answers as negative examples, and filled symbols for prompts using low-rated human answers as negative examples.