High Accuracy, Less Talk (HALT): Reliable LLMs through Capability-Aligned Finetuning

TL;DR

This work proposes post-training an LLM to generate content only when confident in its correctness and to otherwise (partially) abstain, and finetune four open-source models for biography writing, mathematics, coding, and medicine with HALT for three different trade-off thresholds.

Abstract

Large Language Models (LLMs) currently respond to every prompt. However, they can produce incorrect answers when they lack knowledge or capability -- a problem known as hallucination. We instead propose post-training an LLM to generate content only when confident in its correctness and to otherwise (partially) abstain. Specifically, our method, HALT, produces capability-aligned post-training data that encodes what the model can and cannot reliably generate. We generate this data by splitting responses of the pretrained LLM into factual fragments (atomic statements or reasoning steps), and use ground truth information to identify incorrect fragments. We achieve capability-aligned finetuning responses by either removing incorrect fragments or replacing them with "Unsure from Here" -- according to a tunable threshold that allows practitioners to trade off response completeness and mean correctness of the response's fragments. We finetune four open-source models for biography writing, mathematics, coding, and medicine with HALT for three different trade-off thresholds. HALT effectively trades off response completeness for correctness, increasing the mean correctness of response fragments by 15% on average, while resulting in a 4% improvement in the F1 score (mean of completeness and correctness of the response) compared to the relevant baselines. By tuning HALT for highest correctness, we train a single reliable Llama3-70B model with correctness increased from 51% to 87% across all four domains while maintaining 53% of the response completeness achieved with standard finetuning.

Figures (11)

• Figure 1: Comparison between Standard and HALT Finetuning for LLMs: Standard finetuning trains the LLM on responses that potentially exceed the pretrained LLM's capabilities, which results in incorrect outputs at test time. HALT finetuning trains the model only on content within the pretrained LLM's capability limits, omitting unknowns or replacing them with "Unsure from here". HALT finetuning improves response correctness as the LLM is trained to generate responses according to its capabilities, which may result in partially incomplete responses when the pretrained LLM's capabilities are insufficient.
• Figure 2: The $\mathrm{HALT}$ pipeline includes generating a preliminary response via few-shot prompting of the pretrained LLM, splitting it into factual fragments, and assessing each fragment's correctness before compiling the HALT finetuning response. For a causally dependent response, as displayed at the top, an error results in an "Unsure from here" marker to indicate uncertainty. For a response consisting of independent fragments, as shown at the bottom, incorrect fragments are removed.
• Figure 3: Response correctness (y-axis) and completeness (x-axis) for LLama3-70B (top) and Llama3-8B (bottom) when finetuned with different methods (desired number of fragments is $n_\textrm{all}$). F1 Score is constant along curved grey lines, and highest in the top right corner. For HALT and Randomtrim, results are shown with different trade-offs between completeness and correctness, where lighter colors indicate tuning for higher correctness. We omit results with less than 5% completeness, e.g., RandomTrim results for Wikipedia Biographies and Math. We observe that $\mathrm{HALT}$ allows for strongly influencing the trade-off between correctness and completeness across all four datasets while achieving higher F1 scores (closer to the top right corner) than baseline methods. We show results for Gemma2-9B and Mistral7B in Figure \ref{['fig:gemma-mistral-mainresults']} in the Appendix.
• Figure 4: Example responses (ground truth response in the top left corner), with completeness, correctness, and F1 scores annotated for each. The number of required fragments here is three. We observe that the highest F1 score is achieved when the LLM answers with the correct first fragments and abstains after.
• Figure 5: We show results for training a single reliable LLama3-70B model with $\mathrm{HALT}$ tuned for increased correctness, trained on equal shares of all four datasets, referred to HALT (Combined Model). We additionally plot results for HALT trained on each dataset individually, referred to as HALT, and results when finetuning on the Unchanged dataset. We observe that $\mathrm{HALT}$ allows for increasing average accuracy across all datasets by 36% to 87% while maintaining an average completeness of 25%.