Selective Prompting Tuning for Personalized Conversations with LLMs

TL;DR

This work initializes a set of soft prompts and uses a trainable dense retriever to adaptively select suitable soft prompts for LLMs according to different input contexts, where the prompt retriever is dynamically updated through feedback from the LLMs.

Abstract

In conversational AI, personalizing dialogues with persona profiles and contextual understanding is essential. Despite large language models' (LLMs) improved response coherence, effective persona integration remains a challenge. In this work, we first study two common approaches for personalizing LLMs: textual prompting and direct fine-tuning. We observed that textual prompting often struggles to yield responses that are similar to the ground truths in datasets, while direct fine-tuning tends to produce repetitive or overly generic replies. To alleviate those issues, we propose \textbf{S}elective \textbf{P}rompt \textbf{T}uning (SPT), which softly prompts LLMs for personalized conversations in a selective way. Concretely, SPT initializes a set of soft prompts and uses a trainable dense retriever to adaptively select suitable soft prompts for LLMs according to different input contexts, where the prompt retriever is dynamically updated through feedback from the LLMs. Additionally, we propose context-prompt contrastive learning and prompt fusion learning to encourage the SPT to enhance the diversity of personalized conversations. Experiments on the CONVAI2 dataset demonstrate that SPT significantly enhances response diversity by up to 90\%, along with improvements in other critical performance indicators. Those results highlight the efficacy of SPT in fostering engaging and personalized dialogue generation. The SPT model code (https://github.com/hqsiswiliam/SPT) is publicly available for further exploration.

Figures (4)

• Figure 1: Selective Prompt Tuning (SPT) process for personalized dialogue generation with large language models (LLMs). The process starts by computing similarity scores for $K$ soft prompts given the context, followed by LLM loss computation. The prompts are then fed into the LLM along with the context to generate multiple LLM losses which are normalized. A dense retriever computes another set of scores for a different context to inform the contrastive loss. The four computed losses guide the updates to the soft prompts and the retriever to enhance response diversity and relevance.
• Figure 2: Analysis of the usage of each soft prompt cross the conversational process, where the horizontal axis represents the index of the conversational turn and the vertical axis denotes the times that each soft prompt is chosen.
• Figure 3: The varied response styles of the Llama2-7B-SPT model, highlighting its tendency to incorporate emojis into responses during initial conversational turns.
• Figure 4: Four case studies, where PT denotes the prompt tuning method prompt-tuning.