Steering Conversational Large Language Models for Long Emotional Support Conversations

TL;DR

The paper tackles the challenge of steering large language models in long emotional support conversations by introducing the Strategy Relevant Attention (SRA) metric to quantify alignment between model attention and strategy prompts. It provides an extended ESConv-based synthetic dataset with multiple strategy-conditioned continuations and demonstrates that LoRA fine-tuning of Llama2-7B-chat and Llama3-8B-instruct yields substantially higher strategy adherence and SRA than baselines, validated by both automated metrics and human judgments. A RoBERTa-large strategy classifier enables automatic evaluation of adherence, achieving high accuracy on held-out data. Collectively, the approach shows that strategic prompting and targeted fine-tuning can improve steerability in long-horizon emotional support interactions while preserving coherence and naturalness, with implications for safer and more controllable conversational agents.

Abstract

In this study, we address the challenge of enabling large language models (LLMs) to consistently adhere to emotional support strategies in extended conversations. We focus on the steerability of the Llama-2 and Llama-3 suite of models, examining their ability to maintain these strategies throughout interactions. To assess this, we introduce the Strategy Relevant Attention (SRA) metric, which quantifies the model's adherence to the prompted strategy through attention maps. To facilitate our study, we create a strategy-conditioned synthetic conversational dataset derived from the ESConv dataset. We also propose various baselines informed by our proposed SRA metric to address the challenge and propose a fine-tuned model that significantly enhances the steerability of the base model in following the strategy throughout the conversation. The code and data are publicly available on our GitHub.

Figures (11)

• Figure 1: A sample continuation of a conversation using "Provide Different Perspectives" strategy, given by three different prompt templates sorted by the SRA metric increasing from bottom to top using Llama-70b-chat model. The model output using the prompt template with higher SRA adheres better to the given strategy.
• Figure 2: Left: average accuracy of the strategy following for each model with respect to the turn of the conversation, Right: average SRA of the responses with respect to the turn of the conversation
• Figure 3: Six experimental prompt templates to measure SRA with respect to the position of strategy guidelines inside the prompt.
• Figure 4: Correlation of the SRA metric with the accuracy of the strategy following for llama2-7b-chat fine-tuned model vs. baselines. Y-axis is in logarithmic scale.
• Figure 5: y-axis shows the normalized score of the annotators for each annotation task and x-axis shows the normalized log of difference between responses in the annotation task.