Eliciting Behaviors in Multi-Turn Conversations

TL;DR

This paper tackles eliciting specific behaviors from LLMs in multi-turn conversations by formalizing a threefold taxonomy of elicitation methods (prior knowledge, offline interaction, online interaction) and introducing EMBER, a generalized online, multi-turn reinforcement-learning framework. Through experiments on three task benchmarks, EMBER consistently achieves higher behavior-elicitation success with far fewer queries than static or offline methods, highlighting the efficiency of online approaches while exposing the limits of static benchmarks which saturate as models evolve. The work demonstrates that dynamic, adaptive test-case generation is crucial for robust multi-turn evaluation, and provides detailed ablation studies showing benefits of high-level strategy decomposition and careful system prompts. Collectively, these results advocate shifting the community toward adaptive benchmarks and scalable, interactive elicitation to better diagnose and improve real-world conversational AI systems.

Abstract

Identifying specific and often complex behaviors from large language models (LLMs) in conversational settings is crucial for their evaluation. Recent work proposes novel techniques to find natural language prompts that induce specific behaviors from a target model, yet they are mainly studied in single-turn settings. In this work, we study behavior elicitation in the context of multi-turn conversations. We first offer an analytical framework that categorizes existing methods into three families based on their interactions with the target model: those that use only prior knowledge, those that use offline interactions, and those that learn from online interactions. We then introduce a generalized multi-turn formulation of the online method, unifying single-turn and multi-turn elicitation. We evaluate all three families of methods on automatically generating multi-turn test cases. We investigate the efficiency of these approaches by analyzing the trade-off between the query budget, i.e., the number of interactions with the target model, and the success rate, i.e., the discovery rate of behavior-eliciting inputs. We find that online methods can achieve an average success rate of 45/19/77% with just a few thousand queries over three tasks where static methods from existing multi-turn conversation benchmarks find few or even no failure cases. Our work highlights a novel application of behavior elicitation methods in multi-turn conversation evaluation and the need for the community to move towards dynamic benchmarks.

Figures (8)

• Figure 1: Saturation of a static benchmark: Static tests from MT-Bench-101 self-affirmation task released in February 2024 are saturated by models released within a year, whereas online methods can still find failures efficiently in newer models.
• Figure 1: Success rate of methods on self-affirmation and inference memory. For methods involving training, we report the mean and standard deviation over three seeds. Overall, online methods have the highest success rate.
• Figure 2: Three families of elicitation methods. We categorize elicitation methods based on how they interact with the target model: prior knowledge only, offline interactions, and online interactions.
• Figure 3: An overview of EMBER: A multi-turn behavior elicitation method using online RL. At each turn, the policy model first takes in a conversation context, including a system prompt specifying the test objective and the first $i-1$ turns, and generates the $i$th policy turn by sampling from $\mathcal{D}_\text{online}(x)$. Then, we compute the rewards by scoring the generated policy turn, along with the conversation context and the target model outputs, with the rubrics.
• Figure 4: Query efficiency of different methods. Color represents the method family. Orange: Prior. Blue: Offline. Green: Online. Shape represents the task. In general, we observe a trade-off between the success rate and #queries to the target model.