EAVIT: Efficient and Accurate Human Value Identification from Text data via LLMs
Wenhao Zhu, Yuhang Xie, Guojie Song, Xin Zhang
TL;DR
The paper addresses the challenge of identifying human values from text using large language models, which is hindered by long contextual definitions and high inference costs. It introduces EAVIT, a hybrid framework that fuses a locally finetuned value detector with online LLMs, augmented by explanation-based fine-tuning, diverse data generation, and a candidate-value sampling strategy to keep inputs concise. Through extensive experiments on public datasets (ValueNet Augmented, Webis-ArgValues-22, Touché23-ValueEval), EAVIT achieves state-of-the-art accuracy while reducing LLM token usage to about one-sixth of what direct prompting requires, especially improving performance on infrequent values. A case study on virtual individuals demonstrates the potential for passive, text-based value inference aligned with psychological measures, highlighting practical implications for LLM alignment and computational psychology. The approach offers a scalable, cost-efficient solution for large-scale value identification and provides a rich set of prompts and templates to support reproducibility and future work.
Abstract
The rapid evolution of large language models (LLMs) has revolutionized various fields, including the identification and discovery of human values within text data. While traditional NLP models, such as BERT, have been employed for this task, their ability to represent textual data is significantly outperformed by emerging LLMs like GPTs. However, the performance of online LLMs often degrades when handling long contexts required for value identification, which also incurs substantial computational costs. To address these challenges, we propose EAVIT, an efficient and accurate framework for human value identification that combines the strengths of both locally fine-tunable and online black-box LLMs. Our framework employs a value detector - a small, local language model - to generate initial value estimations. These estimations are then used to construct concise input prompts for online LLMs, enabling accurate final value identification. To train the value detector, we introduce explanation-based training and data generation techniques specifically tailored for value identification, alongside sampling strategies to optimize the brevity of LLM input prompts. Our approach effectively reduces the number of input tokens by up to 1/6 compared to directly querying online LLMs, while consistently outperforming traditional NLP methods and other LLM-based strategies.