Intuitive or Dependent? Investigating LLMs' Behavior Style to Conflicting Prompts

TL;DR

This work investigates how LLMs behave when facing conflicts between prompts and internal memory, with implications for retrieval-augmented generation. It introduces the Knowledge Robustness Evaluation (KRE) dataset and a five-part evaluation pipeline to measure memory reliance, factual robustness, and decision-making styles, augmented by role-play interventions. The study quantifies three decision-making styles—Dependent, Intuitive, and Rational/irrational—via the Decision-Making Style Score (DMSS) and demonstrates that role-play can shift models’ styles and robustness, with adaptivity varying across models. Key findings show that many models privilege external prompts (dependent style) or internal memory (intuitive style), while rational styles emerge in stronger models like GPT-4 and Bard; however, robustness to misleading prompts remains a critical challenge. The results offer practical guidance for designing prompts and choosing models or interventions (e.g., dynamic role-play) to optimize performance in conflict scenarios and outline directions for improving factual knowledge integration and memory-based reasoning in LLMs.

Abstract

This study investigates the behaviors of Large Language Models (LLMs) when faced with conflicting prompts versus their internal memory. This will not only help to understand LLMs' decision mechanism but also benefit real-world applications, such as retrieval-augmented generation (RAG). Drawing on cognitive theory, we target the first scenario of decision-making styles where there is no superiority in the conflict and categorize LLMs' preference into dependent, intuitive, and rational/irrational styles. Another scenario of factual robustness considers the correctness of prompt and memory in knowledge-intensive tasks, which can also distinguish if LLMs behave rationally or irrationally in the first scenario. To quantify them, we establish a complete benchmarking framework including a dataset, a robustness evaluation pipeline, and corresponding metrics. Extensive experiments with seven LLMs reveal their varying behaviors. And, with role play intervention, we can change the styles, but different models present distinct adaptivity and upper-bound. One of our key takeaways is to optimize models or the prompts according to the identified style. For instance, RAG models with high role play adaptability may dynamically adjust the interventions according to the quality of retrieval results -- being dependent to better leverage informative context; and, being intuitive when external prompt is noisy.

Figures (11)

• Figure 1: In conflict situation , LLMs may depend on the prompt or intuitively answer based on memory.
• Figure 2: The pipeline incorporates several steps to assess the robustness of LLMs: 1. memory assessment in Section \ref{['subsection_knowledge Assessment']}. 2. Factual robustness evaluation in Section \ref{['subsection_knowledge context']}. 3. Few-shot example influence in Section \ref{['few-shot exampple influence']}. 4. Decision-making style analysis in Sec \ref{['descion analysis']}. 5. Role play intervention and leaderboard in Sec \ref{['rol play intervention']}
• Figure 3: The VR score (%) and The RR score (%) for model ChatGPT and Vicuna-13B.
• Figure 4: RR and VR of ChatGPT and Vicuna under instruction with and without hint (Sec \ref{['kre construction']}). The corresponding number of negative answers and invalid responses.
• Figure 5: The VR and RR score (%) under the influence of three few-shot configurations.