DELIA: Diversity-Enhanced Learning for Instruction Adaptation in Large Language Models
Yuanhao Zeng, Fei Ren, Xinpeng Zhou, Yihang Wang, Yingxia Shao
TL;DR
DELIA addresses a core limitation of instruction tuning in large language models by leveraging diversity-rich data synthesis to transform biased instruction-tuning features into approximations of ideal task features, without explicit prior knowledge of those ideals. By exploiting the buffering effect of extensive diverse data, DELIA enables data-driven, scalable learning and better aligns internal representations with downstream semantics. Empirical results show significant gains over standard instruction tuning and baselines, including Icelandic-English translation BLEURT improvements of 17.07%-33.41% on WMT-21 gemma-7b-it and a 36.1% accuracy boost on formatted text generation with Llama2-7b-chat, along with unique alignment of new token semantics to prior meanings. Overall, DELIA demonstrates a practical, data-centered approach to instruction adaptation that enhances cross-task transfer in LLMs and narrows gaps between instruction formats and underlying semantic understanding.
Abstract
Although instruction tuning is widely used to adjust behavior in Large Language Models (LLMs), extensive empirical evidence and research indicates that it is primarily a process where the model fits to specific task formats, rather than acquiring new knowledge or capabilities. We propose that this limitation stems from biased features learned during instruction tuning, which differ from ideal task-specfic features, leading to learn less underlying semantics in downstream tasks. However, ideal features are unknown and incalculable, constraining past work to rely on prior knowledge to assist reasoning or training, which limits LLMs' capabilities to the developers' abilities, rather than data-driven scalable learning. In our paper, through our novel data synthesis method, DELIA (Diversity-Enhanced Learning for Instruction Adaptation), we leverage the buffering effect of extensive diverse data in LLMs training to transform biased features in instruction tuning into approximations of ideal features, without explicit prior ideal features. Experiments show DELIA's better performance compared to common instruction tuning and other baselines. It outperforms common instruction tuning by 17.07%-33.41% on Icelandic-English translation bleurt score (WMT-21 dataset, gemma-7b-it) and improves accuracy by 36.1% on formatted text generation (Llama2-7b-chat). Notably, among knowledge injection methods we've known, DELIA uniquely align the internal representations of new special tokens with their prior semantics.