Latent Distance Guided Alignment Training for Large Language Models

TL;DR

LD-Align introduces an annotation-free alignment framework for large language models by leveraging latent-space distances learned through an auto-encoder to guide direct preference optimization. Real SFT responses are treated as winners and model-generated responses as losers; a latent-distance metric informs a weighting scheme within the DPO objective, emphasizing poorly aligned instances while avoiding overfitting. Experiments on Ultrachat-200k with Mistral-7B/Zephyr baselines show LD-Align yields consistent improvements across truthfulness, reasoning, and multi-round dialogue benchmarks, achieving roughly a 6% average gain over competing methods. This approach demonstrates that latent reconstruction signals can substitute external annotations for effective alignment, offering a cost-efficient and scalable path to better human-aligned LLM behavior.

Abstract

Ensuring alignment with human preferences is a crucial characteristic of large language models (LLMs). Presently, the primary alignment methods, RLHF and DPO, require extensive human annotation, which is expensive despite their efficacy. The significant expenses associated with current alignment techniques motivate researchers to investigate the development of annotation-free alignment training methods. In pursuit of improved alignment without relying on external annotation, we introduce Latent Distance Guided Alignment Training (LD-Align). This approach seeks to align the model with a high-quality supervised fine-tune dataset using guidance from a latent space. The latent space is generated through sample reconstruction, akin to auto-encoding. Consequently, we utilize the distance between sample pairs in the latent space to guide DPO-based alignment training. Extensive experimentation and evaluation show the efficacy of our proposed method in achieving notable alignment.