Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Models

Zikai Xie

TL;DR

This work investigates hallucinations in large language models and the impact of the order in which reasoning and answers are produced. It introduces Reasoning Order as Benchmark to quantify model consistency across answer-first and logic-first prompts, and Reflexive Prompting, a two-step prompting strategy that leverages both outputs to improve reliability without retraining. Empirical results across multiple datasets and models show that reflexive prompting often yields higher accuracy and that consistency metrics correlate with performance, supporting the method as a practical diagnostic and improvement tool. The findings offer a cost-efficient approach to enhance LLM reliability in high-stakes settings and suggest directions for future decoder-level or inference-time enhancements.

Abstract

Large language models (LLMs) have generated significant attention since their inception, finding applications across various academic and industrial domains. However, these models often suffer from the "hallucination problem", where outputs, though grammatically and logically coherent, lack factual accuracy or are entirely fabricated. A particularly troubling issue discovered and widely discussed recently is the numerical comparison error where multiple LLMs incorrectly infer that "9.11$>$9.9". We discovered that the order in which LLMs generate answers and reasoning impacts their consistency. Specifically, results vary significantly when an LLM generates an answer first and then provides the reasoning versus generating the reasoning process first and then the conclusion. Inspired by this, we propose a new benchmark method for assessing LLM consistency: comparing responses generated through these two different approaches. This benchmark effectively identifies instances where LLMs fabricate answers and subsequently generate justifications. Furthermore, we introduce a novel and straightforward prompt strategy designed to mitigate this issue. Experimental results demonstrate that this strategy improves performance across various LLMs compared to direct questioning. This work not only sheds light on a critical flaw in LLMs but also offers a practical solution to enhance their reliability.

Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Models

TL;DR

This work investigates hallucinations in large language models and the impact of the order in which reasoning and answers are produced. It introduces Reasoning Order as Benchmark to quantify model consistency across answer-first and logic-first prompts, and Reflexive Prompting, a two-step prompting strategy that leverages both outputs to improve reliability without retraining. Empirical results across multiple datasets and models show that reflexive prompting often yields higher accuracy and that consistency metrics correlate with performance, supporting the method as a practical diagnostic and improvement tool. The findings offer a cost-efficient approach to enhance LLM reliability in high-stakes settings and suggest directions for future decoder-level or inference-time enhancements.

Abstract

Large language models (LLMs) have generated significant attention since their inception, finding applications across various academic and industrial domains. However, these models often suffer from the "hallucination problem", where outputs, though grammatically and logically coherent, lack factual accuracy or are entirely fabricated. A particularly troubling issue discovered and widely discussed recently is the numerical comparison error where multiple LLMs incorrectly infer that "9.119.9". We discovered that the order in which LLMs generate answers and reasoning impacts their consistency. Specifically, results vary significantly when an LLM generates an answer first and then provides the reasoning versus generating the reasoning process first and then the conclusion. Inspired by this, we propose a new benchmark method for assessing LLM consistency: comparing responses generated through these two different approaches. This benchmark effectively identifies instances where LLMs fabricate answers and subsequently generate justifications. Furthermore, we introduce a novel and straightforward prompt strategy designed to mitigate this issue. Experimental results demonstrate that this strategy improves performance across various LLMs compared to direct questioning. This work not only sheds light on a critical flaw in LLMs but also offers a practical solution to enhance their reliability.
Paper Structure (15 sections, 4 figures, 4 tables, 2 algorithms)

This paper contains 15 sections, 4 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Hallucination Problem
  4. Think Before You Speak
  5. Prompting Methods for Hallucination
  6. Methods
  7. Reasoning Order as Benchmark
  8. Reflexive Prompting
  9. Experiments
  10. Preliminary Experiment
  11. Experimental Settings
  12. Experimental Results
  13. Conclusion
  14. Limitations
  15. Case Study Example

Figures (4)

  • Figure 1: These two figures show reproduction of the "9.11$>$9.9" error in major LLMs (GPT-4o above, Llama3 below), where the models incorrectly gives out the wrong answer first and defend it afterwards.
  • Figure 2: Comparison of the "answer first, logic second" prompt and original prompt using Claude. Result shows that incorrect answer is given when asked to change the output order, although the model can correctly reason and answer the question under the other order.
  • Figure A.1: Case study example 1, using question No. 52 in TruthfulQA on Claude that answer-first result is wrong while logic-first result is correct.
  • Figure A.2: Case study example 2, using question No. 130 in TruthfulQA on Claude that answer-first result is correct while logic-first result is wrong.