Trustful LLMs: Customizing and Grounding Text Generation with Knowledge Bases and Dual Decoders
Xiaofeng Zhu, Jaya Krishna Mandivarapu
TL;DR
The paper tackles the problem of domain-groundedness and hallucinations in LLM copilots by introducing two complementary strategies: a post-processing hallucination-correction (HC) method that leverages knowledge triplets from RAG context to verify and adjust generated content, and a TrustfulLLM dual-decoder architecture that fuses guided context with the user prompt through shared weights and cross-attention. The HC approach constructs knowledge-graph representations of both the guided context and the LLM output, then iteratively replaces or prunes triplets to produce a verified output $Y^*$, reducing misaligned content without expanding the graph. The TrustfulLLM model integrates a guided-context decoder and a prompt decoder with CROSSATTN$(H_p,H_g)$, enabling grounded generation while maintaining fluency, with the guided context sourced from RAG results and augmented during training. Empirical results on a Microsoft M365-derived dataset show that combining HC with TrustfulLLM yields the best performance across groundedness, semantic similarity, and retrieval-grounded metrics, supporting practical deployment in domain-specific copilots and highlighting considerations for production constraints such as intent detection, red-teaming, and privacy. Overall, the work demonstrates that grounding by design—through both post-hoc correction and grounded generation—can substantially improve the reliability and safety of domain-adapted LLM systems.
Abstract
Although people are impressed by the content generation skills of large language models, the use of LLMs, such as ChatGPT, is limited by the domain grounding of the content. The correctness and groundedness of the generated content need to be based on a verified context, such as results from Retrieval-Augmented Generation (RAG). One important issue when adapting LLMs to a customized domain is that the generated responses are often incomplete, or the additions are not verified and may even be hallucinated. Prior studies on hallucination detection have focused on evaluation metrics, which are not easily adaptable to dynamic domains and can be vulnerable to attacks like jail-breaking. In this work, we propose 1) a post-processing algorithm that leverages knowledge triplets in RAG context to correct hallucinations and 2) a dual-decoder model that fuses RAG context to guide the generation process.