Is Length Really A Liability? An Evaluation of Multi-turn LLM Conversations using BoolQ

TL;DR

The study addresses whether longer LLM conversations undermine veracity by moving beyond single-prompt benchmarks to a controlled multi-turn evaluation on an enriched BoolQ dataset. It combines four scaffolds and five conversation lengths across three instruction-tuned LLMs, analyzing outcomes with multinomial logistic regression to capture accuracy and abstention patterns. The key contribution is the demonstration that length effects are model- and scaffold-specific, uncovering vulnerabilities such as confident misinformation in Qwen2.5 under certain prompts that static benchmarks would miss, and highlighting the importance of confidence calibration in safety assessments. The findings suggest deployment strategies should tailor safety considerations to model architecture and interaction context, and they call for broader, more nuanced evaluation frameworks to better anticipate real-world harms.

Abstract

Single-prompt evaluations dominate current LLM benchmarking, yet they fail to capture the conversational dynamics where real-world harm occurs. In this study, we examined whether conversation length affects response veracity by evaluating LLM performance on the BoolQ dataset under varying length and scaffolding conditions. Our results across three distinct LLMs revealed model-specific vulnerabilities that are invisible under single-turn testing. The length-dependent and scaffold-specific effects we observed demonstrate a fundamental limitation of static evaluations, as deployment-relevant vulnerabilities could only be spotted in a multi-turn conversational setting.

Figures (1)

• Figure 1: Response patterns for a) Incorrect, b) Correct, and c) IDK across conversation lengths for our target models and four scaffold conditions