Modeling and Predicting Multi-Turn Answer Instability in Large Language Models

TL;DR

The paper addresses the fragility of LLMs under repeated questioning by combining simple multi-turn prompts, Markov chain modeling, and linear probes to quantify and predict answer stability. It demonstrates that model accuracy drifts across turns and can be captured by a two-state Markov process, enabling computation of a stationary accuracy $Acc_{ inf} = \frac{p_{FT}}{p_{TF} + p_{FT}}$ that is typically lower than first-turn performance. Hidden-state analysis via linear probes provides early signals that can predict future answer changes, supporting the idea that robustness can be monitored and potentially mitigated during inference. The findings advocate for stationary accuracy as a robust metric for interactive LLM deployments and highlight the need to improve resistance to prompt-induced instabilities in practical, high-stakes settings.

Abstract

As large language models (LLMs) are adopted in an increasingly wide range of applications, user-model interactions have grown in both frequency and scale. Consequently, research has focused on evaluating the robustness of LLMs, an essential quality for real-world tasks. In this paper, we employ simple multi-turn follow-up prompts to evaluate models' answer changes, model accuracy dynamics across turns with Markov chains, and examine whether linear probes can predict these changes. Our results show significant vulnerabilities in LLM robustness: a simple "Think again" prompt led to an approximate 10% accuracy drop for Gemini 1.5 Flash over nine turns, while combining this prompt with a semantically equivalent reworded question caused a 7.5% drop for Claude 3.5 Haiku. Additionally, we find that model accuracy across turns can be effectively modeled using Markov chains, enabling the prediction of accuracy probabilities over time. This allows for estimation of the model's stationary (long-run) accuracy, which we find to be on average approximately 8% lower than its first-turn accuracy for Gemini 1.5 Flash. Our results from a model's hidden states also reveal evidence that linear probes can help predict future answer changes. Together, these results establish stationary accuracy as a principled robustness metric for interactive settings and expose the fragility of models under repeated questioning. Addressing this instability will be essential for deploying LLMs in high-stakes and interactive settings where consistent reasoning is as important as initial accuracy.

Figures (30)

• Figure 2: Accuracy drift across 10 turns for GPT-4.1-nano and Gemini 1.5 Flash on MathQA questions. For GPT-4.1-nano, the maximum accuracy decline (from the first turn to the lowest-performing turn) for each prompt was $30.3\% \rightarrow 24.6\%$ on TA, $31.6\% \rightarrow 26.8\%$ on RUS, and $31.5\% \rightarrow 19.1\%$ on URW. For Gemini 1.5 Flash, the maximum accuracy decline for each prompt was $40.7\% \rightarrow 30.1\%$ on TA, $41.4\% \rightarrow 36.4\%$ on RUS, and $41.0\% \rightarrow 29.1\%$ on URW.
• Figure 3: Accuracy drift across 6 turns for Claude 3.5 Haiku (MathQA) and GPT-4o (MMLU). For Claude 3.5 Haiku, the maximum accuracy decline for each prompt was $81.6\% \rightarrow 72.5\%$ on TA, $81.2\% \rightarrow 71.5\%$ on RUS, and $80.4\% \rightarrow 50.0\%$ on URW. For GPT-4o, the maximum accuracy decline for each prompt was from $88.6\% \rightarrow 86.1\%$ on TA, from $88.8\% \rightarrow 86.3\%$ on RUS, and from $89.5\% \rightarrow 84.1\%$ on URW.
• Figure 4: True vs. simulated accuracy for GPT-4.1-nano and Gemini 1.5 Flash. Both models were prompted using TA on MathQA questions. For GPT-4.1-nano, accuracies on turn 10 deviated by 0.38%. For Gemini 1.5 Flash, accuracies on turn 10 deviated by 3.76%. The close match between simulated and true accuracy shows that the Markov simulation accurately captures the model's multi-turn dynamics.
• Figure 5: True vs. simulated accuracy for GPT-4o (MMLU) and Claude 3.5 Haiku (MathQA) with the rephrased RUS prompt. For GPT-4o, accuracies on turn 6 deviated by 0.11%. For Claude 3.5 Haiku, accuracies on turn 6 deviated by 3.99%. The close match between simulated and true accuracy shows that the Markov simulation accurately captures the model's multi-turn dynamics.
• Figure 6: Accuracy degradation of Gemini 1.5 Flash on MathQA. Overall, the model’s accuracy decreases by an average of 12.03% for simple follow-up prompts and 3.97% for rephrased prompts. This suggests that a model is more robust to reworded questions than to simple follow-up prompts.