HAL: Inducing Human-likeness in LLMs with Alignment

TL;DR

HAL addresses the challenge of aligning LLMs to conversational human-likeness by deriving an interpretable score from contrastive dialogue data. It computes HL16Q via $HL16Q(A) = sum_{i=1}^{M} A_i W_i + b$ after identifying HL32Q traits, and uses this score as a reward in Direct Preference Optimization to tune models. Across sizes, HAL yields higher perceived human-likeness in human evaluations while largely preserving non-human-likeness benchmarks, and its trait-level interpretability enables inspection of alignment behavior and potential reward hacking. This work demonstrates a general method to measure and align soft qualitative properties in LLMs, with implications for controllable, transparent, and ethically informed AI deployment.

Abstract

Conversational human-likeness plays a central role in human-AI interaction, yet it has remained difficult to define, measure, and optimize. As a result, improvements in human-like behavior are largely driven by scale or broad supervised training, rather than targeted alignment. We introduce Human Aligning LLMs (HAL), a framework for aligning language models to conversational human-likeness using an interpretable, data-driven reward. HAL derives explicit conversational traits from contrastive dialogue data, combines them into a compact scalar score, and uses this score as a transparent reward signal for alignment with standard preference optimization methods. Using this approach, we align models of varying sizes without affecting their overall performance. In large-scale human evaluations, models aligned with HAL are more frequently perceived as human-like in conversation. Because HAL operates over explicit, interpretable traits, it enables inspection of alignment behavior and diagnosis of unintended effects. More broadly, HAL demonstrates how soft, qualitative properties of language--previously outside the scope for alignment--can be made measurable and aligned in an interpretable and explainable way.

Figures (11)

• Figure 1: The HAL pipeline: (a) identifying human-likeness traits from contrastive dialogues (e.g. Turing tests), (b) learning trait weights via a proxy classification task, and (c) computing a human-likeness score for alignment.
• Figure 2: Violin plot of HL16Q Score on Out-of-distribution (OOD) dataset containing Human-AI and Human-Human conversations.
• Figure 3: Human-likeness Score with 95% Confidence Interval (shaded) on 10 epochs of training with DPO.
• Figure 4: Qwen2.5-14B HL16Q individual statement interpetation
• Figure 5: The evaluation interface for Chatbot Arena-style A/B testing.