Implementing Rational Choice Functions with LLMs and Measuring their Alignment with User Preferences

TL;DR

This work addresses how to align LLM-driven decision agents with user preferences over a set of alternatives by generalizing from ranking to rational preferences. It introduces design principles (Pairwise-Score and Pairwise-SCC) and two alignment metrics (SPO for partial alignment and a Kendall-based distance with penalty for full alignment) to implement and evaluate rational choice functions in intelligent user interfaces. An automotive IUI study demonstrates that Pairwise-Score excels in partial alignment while Pairwise-SCC achieves closer full alignment, with performance depending on the LLM and prompt design. The framework provides a principled path toward trustworthy, user-preference-aligned LLM decision-making, while highlighting challenges around prompt sensitivity, model choice, and scalability that guide future work.

Abstract

As large language models (LLMs) become integral to intelligent user interfaces (IUIs), their role as decision-making agents raises critical concerns about alignment. Although extensive research has addressed issues such as factuality, bias, and toxicity, comparatively little attention has been paid to measuring alignment to preferences, i.e., the relative desirability of different alternatives, a concept used in decision making, economics, and social choice theory. However, a reliable decision-making agent makes choices that align well with user preferences. In this paper, we generalize existing methods that exploit LLMs for ranking alternative outcomes by addressing alignment with the broader and more flexible concept of user preferences, which includes both strict preferences and indifference among alternatives. To this end, we put forward design principles for using LLMs to implement rational choice functions, and provide the necessary tools to measure preference satisfaction. We demonstrate the applicability of our approach through an empirical study in a practical application of an IUI in the automotive domain.