Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-stage Prompt Refinement for Mitigating Hallucinations in Large Language Models

Jung-Woo Shim, Yeong-Joon Ju, Ji-Hoon Park, Seong-Whan Lee

TL;DR

This work tackles hallucinations in large language models by focusing on prompt quality. It introduces Multi-stage Prompt Refinement (MPR), a lightweight, modular framework that uses small language models trained with 4-bit QLoRA to clean prompts, fix punctuation and grammar, resolve ambiguities, and generate task-specific descriptions with self-reflection. Across GSM8K, SQuAD, Natural Questions, and well-formed queries, MPR reduces hallucinations and improves output quality across multiple base models, achieving average win-rate improvements above $85\%$ in sabotage tests and showing strong compatibility with post-hoc mitigation methods. The approach is efficient, model-agnostic, and scalable, offering practical benefits for reliable LLM deployment in diverse domains, with acknowledged limitations and directions for future domain-specific and human-in-the-loop enhancements.

Abstract

Recent advancements in large language models (LLMs) have shown strong performance in natural language understanding and generation tasks. However, LLMs continue to encounter challenges with hallucinations, where models generate plausible but incorrect information. While several factors contribute to hallucinations, the impact of ill-formed prompts, prompts with ambiguous wording, incorrect grammar, or incomplete information, was relatively under explored. To address this, we introduce Multi-stage Prompt Refinement (MPR), a framework designed to systematically improve these ill-formed prompts across multiple stages. Each stage addresses specific errors such as punctuation, typographical mistakes, and misuse of key terms, using small language models (SLMs) fine-tuned for these tasks. MPR iteratively enhances the clarity of prompts with additional context and employs a self-reflection mechanism with ranking to prioritize the most relevant input. Experimental results on hallucination benchmarks show that prompts refined by MPR achieve over an 85~\% win rate compared to their original forms, demonstrating its effectiveness in reducing hallucinations and improving LLM output accuracy. Interestingly, we reveal that MPR can be combined with existing post-hoc hallucination mitigation frameworks, further enhancing its versatility. MPR provides a lightweight and adaptable solution for enhancing LLM reliability across various domains.

Multi-stage Prompt Refinement for Mitigating Hallucinations in Large Language Models

TL;DR

This work tackles hallucinations in large language models by focusing on prompt quality. It introduces Multi-stage Prompt Refinement (MPR), a lightweight, modular framework that uses small language models trained with 4-bit QLoRA to clean prompts, fix punctuation and grammar, resolve ambiguities, and generate task-specific descriptions with self-reflection. Across GSM8K, SQuAD, Natural Questions, and well-formed queries, MPR reduces hallucinations and improves output quality across multiple base models, achieving average win-rate improvements above in sabotage tests and showing strong compatibility with post-hoc mitigation methods. The approach is efficient, model-agnostic, and scalable, offering practical benefits for reliable LLM deployment in diverse domains, with acknowledged limitations and directions for future domain-specific and human-in-the-loop enhancements.

Abstract

Recent advancements in large language models (LLMs) have shown strong performance in natural language understanding and generation tasks. However, LLMs continue to encounter challenges with hallucinations, where models generate plausible but incorrect information. While several factors contribute to hallucinations, the impact of ill-formed prompts, prompts with ambiguous wording, incorrect grammar, or incomplete information, was relatively under explored. To address this, we introduce Multi-stage Prompt Refinement (MPR), a framework designed to systematically improve these ill-formed prompts across multiple stages. Each stage addresses specific errors such as punctuation, typographical mistakes, and misuse of key terms, using small language models (SLMs) fine-tuned for these tasks. MPR iteratively enhances the clarity of prompts with additional context and employs a self-reflection mechanism with ranking to prioritize the most relevant input. Experimental results on hallucination benchmarks show that prompts refined by MPR achieve over an 85~\% win rate compared to their original forms, demonstrating its effectiveness in reducing hallucinations and improving LLM output accuracy. Interestingly, we reveal that MPR can be combined with existing post-hoc hallucination mitigation frameworks, further enhancing its versatility. MPR provides a lightweight and adaptable solution for enhancing LLM reliability across various domains.
Paper Structure (21 sections, 3 figures, 9 tables)

This paper contains 21 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Challenges in Hallucination Mitigation
  4. SLMs and Their Advantages
  5. Prompt Refinement
  6. Multi-stage Prompt Refinement
  7. Dataset Construction for Fine-tuning
  8. Training for Prompt Refinement
  9. Multi-stage Prompt Cleaning and Paraphrasing
  10. Experiments
  11. Experimental Settings
  12. Evaluation Dataset
  13. Hardware Settings
  14. Evaluating the Effectiveness of MPR Across Various Models
  15. Evaluation of Prompt Refinement
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Overview of the MPR framework. MPR is designed to systematically refine user prompts before they are processed by LLMs. The framework addresses common errors in prompts such as punctuation mistakes, typographical issues, and misuse of key terms that can lead to inaccurate or suboptimal outputs from LLMs.
  • Figure 2: Visualization of MPR's effectiveness
  • Figure 3: Visualization of the importance of each module of MPR.