HonestLLM: Toward an Honest and Helpful Large Language Model

TL;DR

HonestLLM tackles the tension between honesty and helpfulness in large language models by formalizing a multi-dimensional honesty framework, introducing HoneSet with 930 challenging queries, and proposing two enhancement pathways: a training-free curiosity-driven prompting approach and a two-stage curriculum-inspired fine-tuning using Direct Preference Optimization. Across nine mainstream LLMs, both methods yield substantial gains in honesty and the holistic H^2 metric, with notable cases such as Llama3-8b gaining about 13.7 percentage points in honesty and Mistral-7b gaining about 51.9 percentage points, alongside large $H^2$ improvements. The study also examines cross-task effects and safety, finding modest utility gains in some tasks and improved safety behavior. Overall, the work advances trustworthy LLM deployments by elevating honesty while maintaining practical usefulness, at a feasible computational cost.

Abstract

Large Language Models (LLMs) have achieved remarkable success across various industries due to their exceptional generative capabilities. However, for safe and effective real-world deployments, ensuring honesty and helpfulness is critical. This paper addresses the question: Can we prioritize the helpfulness of LLMs while preserving their honesty? To begin with, we establish exhaustive principles aimed at guaranteeing the honesty of LLM. Additionally, we introduce a novel dataset, referred to as HoneSet, comprising 930 queries spanning six categories meticulously crafted to assess an LLM's capacity for maintaining honesty. Subsequently, we present two approaches to augmenting honesty and helpfulness in LLMs: a training-free enhancement and a fine-tuning-based improvement. The training-free approach, which is based on curiosity-driven prompting, empowers LLMs to articulate internal confusion and uncertainty regarding queries, thereby optimizing their responses. Conversely, the fine-tuning-based method employs a two-stage process inspired by curriculum learning: initially instructing LLMs to discern between honest and dishonest responses, then refining their training to enhance helpfulness. Experiments conducted on nine prominent LLMs demonstrate a significant improvement in alignment with honesty across all models through the implementation of our proposed enhancements. Particularly noteworthy is the 65.3% enhancement observed in Llama3-8b and the remarkable 124.7% improvement in Mistral-7b, as measured by the H$^{2}$ (honest and helpful) assessment. We believe that our work can pave the way for developing more trustworthy LLMs for real-world applications.

Figures (17)

• Figure 1: (a) The PCA abdi2010principal visualization of honesty-related (top) and harm-related (bottom) hidden state of top layer embeddings extracted from the final token in Llama2-7b's outputs. The harm-related queries come from the previous study zheng2024promptdriven. (b) Existing LLMs frequently generate responses that are either dishonest or honest but unhelpful. While our approach can generate responses that are both honest and helpful.
• Figure 2: Different categories in HoneSet.
• Figure 3: The overall pipeline incorporates both training-free and fine-tuning methods to ensure honesty and enhance helpfulness simultaneously.
• Figure 4: Comprehensive evaluation results of the training-free method.
• Figure 5: Overall score and honesty rates of Llama3-8b and Mistral-7b under different thresholds.