Hallucination-aware Optimization for Large Language Model-empowered Communications

TL;DR

This paper addresses the reliability challenge of hallucinations in large language models applied to telecommunications. It offers a lifecycle-based analysis of hallucination causes, surveys model- and system-based mitigation strategies, and evaluates existing LLM-empowered communication schemes. The authors present a telecom-oriented case study that combines Direct Preference Optimization fine-tuning with a mobile-edge mixture-of-experts to reduce hallucinations and optimize quality of experience, achieving a 20.6% improvement in correctness and substantial QoE gains. The work provides a public telecom hallucination dataset (TeleQnA) and advocates a hybrid approach to enable trustworthy LLM-enabled telecom services, outlining future security, customization, and reasoning directions.

Abstract

Large Language Models (LLMs) have significantly advanced communications fields, such as Telecom Q\&A, mathematical modeling, and coding. However, LLMs encounter an inherent issue known as hallucination, i.e., generating fact-conflicting or irrelevant content. This problem critically undermines the applicability of LLMs in communication systems yet has not been systematically explored. Hence, this paper provides a comprehensive review of LLM applications in communications, with a particular emphasis on hallucination mitigation. Specifically, we analyze hallucination causes and summarize hallucination mitigation strategies from both model- and system-based perspectives. Afterward, we review representative LLM-empowered communication schemes, detailing potential hallucination scenarios and comparing the mitigation strategies they adopted. Finally, we present a case study of a Telecom-oriented LLM that utilizes a novel hybrid approach to enhance the hallucination-aware service experience. On the model side, we publish a Telecom hallucination dataset and apply direct preference optimization to fine-tune LLMs, resulting in a 20.6\% correct rate improvement. Moreover, we construct a mobile-edge mixture-of-experts architecture for optimal LLM expert activation. Our research aims to propel the field of LLM-empowered communications forward by detecting and minimizing hallucination impacts.

Figures (5)

• Figure 1: Practical examples of LLM hallucination. (A) is generated by Llama-7b Chat. (B) and (C) are generated by Claude 3.5 Sonnet.
• Figure 2: The illustration of LLM lifecycles and the causes of hallucinations. Also, the advantages & disadvantages of representative model- and system-based hallucination mitigation strategies are analyzed.
• Figure 3: The data structure of TeleQnA and hallucination detection datasets and the illustration of the DPO training process. Specifically, DPO assigns chosen answers higher rewards than rejected ones. Then, the LoRA-enhanced LLM is trained by maximizing the gap of accumulated reward between itself and the original reference LLM.
• Figure 4: The diffusion DRL-based gating network (left) and the MoE architecture for maximizing hallucination-aware QoE (right). The expert performance in the table is acquired on a workstation with an NVIDIA A6000 GPU. $N_1$ in the left part refers to the number of queries in the user prompt.
• Figure 5: Experimental results. (A): The trends of loss and logits-chosen of DPO training. (B): The training curves of the MoE gating network. (C): The comparison of hallucination-aware QoE. Note that Edge- and Mobile-only refer to merely activating edge and mobile LLM experts, respectively. Round-robin means the experts take turns to serve users.