Learning from Contrastive Prompts: Automated Optimization and Adaptation

TL;DR

The Learning from Contrastive Prompts (LCP) framework is proposed, which offers a systematic approach to prompt engineering, reducing manual effort in deploying LLMs across varied contexts and demonstrates strong adaptability across different model versions, families, and languages.

Abstract

As LLMs evolve, significant effort is spent on manually crafting prompts. While existing prompt optimization methods automate this process, they rely solely on learning from incorrect samples, leading to a sub-optimal performance. Additionally, an unexplored challenge in the literature is prompts effective for prior models may not perform well on newer versions or different languages. We propose the Learning from Contrastive Prompts (LCP) framework to address these gaps, enhancing both prompt optimization and adaptation. LCP employs contrastive learning to generate effective prompts by analyzing patterns in good and bad prompt examples. Our evaluation on the Big-Bench Hard dataset shows that LCP has a win rate of over 76% over existing methods in prompt optimization and demonstrates strong adaptability across different model versions, families, and languages. LCP offers a systematic approach to prompt engineering, reducing manual effort in deploying LLMs across varied contexts.

Figures (6)

• Figure 1: Learning from Contrastive Prompts (LCP) framework. Given an initial prompt and a small training set, LCP generates multiple prompt candidates derived from summaries of common failure reasons for different sample combinations. It leverages the inherent capabilities of LLMs to understand the underlying patterns through contrastive prompts to generate a new prompt.
• Figure 2: Data sampling strategy for prompt adaptation: Run inference on the training set using the source and target LLMs. Identify correctly and incorrectly predicted samples for each model. Select samples correctly predicted by the prior model but incorrectly predicted by the current model.
• Figure 3: Ablation study with number of prompt candidates ($N$) on the left and Effect of contrastive learning (w/ and w/o constrastive learning) on the right. Reported are win rates on the prompt selected with best training set performance (best). AMAR refers to Algorithmic and Arithmetic, NLU to Natural Language Understanding, UWK to Understanding of World Knowledge, and MKR to Multilingual Knowledge and Reasoning categories.
• Figure 4: Accuracy as a function of number of training examples for LCP, AutoHint, and OPRO.
• Figure 5: Accuracy as a function of number of training examples for LCP, AutoHint, and OPRO.