Asking Again and Again: Exploring LLM Robustness to Repeated Questions

TL;DR

This work examines whether repeating questions within a prompt affects large language model QA performance across five models and three reading-comprehension datasets. Four prompt configurations—Open-Book, Closed-Book, QCQ, and Paraphrasing—are tested with repetition levels of 1, 3, and 5, totaling about 90k questions. Repetition yields up to a $6\%$ improvement in some cases but is not statistically significant across configurations, datasets, or models, suggesting robustness to repeated input structures. The findings inform prompt design by indicating that repetition alone does not meaningfully enhance output quality, while highlighting the nuanced interplay between prompt structure, model size, and dataset difficulty.

Abstract

This study investigates whether repeating questions within prompts influences the performance of large language models (LLMs). We hypothesize that reiterating a question within a single prompt might enhance the model's focus on key elements of the query. We evaluate five recent LLMs -- including GPT-4o-mini, DeepSeek-V3, and smaller open-source models -- on three reading comprehension datasets under different prompt settings, varying question repetition levels (1, 3, or 5 times per prompt). Our results demonstrate that question repetition can increase models' accuracy by up to $6\%$. However, across all models, settings, and datasets, we do not find the result statistically significant. These findings provide insights into prompt design and LLM behavior, suggesting that repetition alone does not significantly impact output quality.

Figures (5)

• Figure 1: Overview of our study on question repetition in LLM prompts. We evaluate five recent LLMs across three datasets and four prompt settings, varying question repetition (1, 3, or 5 times) to assess its impact on performance.
• Figure 2: Average accuracy by configuration and repetition level across all models and datasets.
• Figure 3: Average accuracy by model and repetition level across all configurations and datasets.
• Figure 4: Average accuracy by configuration and model across all datasets and repetition levels.
• Figure 5: Average accuracy by dataset and model across all configurations and repetition levels.