How Individual Traits and Language Styles Shape Preferences In Open-ended User-LLM Interaction: A Preliminary Study

TL;DR

Problem addressed: how individual traits and LLM language style shape user preferences in open-ended interaction. Approach: two studies—Study 1 analyzes real-world preference datasets to quantify style effects across populations, and Study 2 uses a within-subject experiment with style manipulation, trait measures, a zero-shot style-transfer pipeline, and Gibbs sampling to elicit preferred styles. Findings: language style influences user preferences in a population-dependent manner, and several traits (e.g., Agreeableness, Extraversion, Neuroticism, Openness, Trust) moderate these effects in distinct ways. Implications: results inform design of LLM interactions to improve user experience while acknowledging risks such as misinformation susceptibility and psychosocial impacts, with planned work to broaden demographics and explore causal relationships.

Abstract

What makes an interaction with the LLM more preferable for the user? While it is intuitive to assume that information accuracy in the LLM's responses would be one of the influential variables, recent studies have found that inaccurate LLM's responses could still be preferable when they are perceived to be more authoritative, certain, well-articulated, or simply verbose. These variables interestingly fall under the broader category of language style, implying that the style in the LLM's responses might meaningfully influence users' preferences. This hypothesized dynamic could have double-edged consequences: enhancing the overall user experience while simultaneously increasing their susceptibility to risks such as LLM's misinformation or hallucinations. In this short paper, we present our preliminary studies in exploring this subject. Through a series of exploratory and experimental user studies, we found that LLM's language style does indeed influence user's preferences, but how and which language styles influence the preference varied across different user populations, and more interestingly, moderated by the user's very own individual traits. As a preliminary work, the findings in our studies should be interpreted with caution, particularly given the limitations in our samples, which still need wider demographic diversity and larger sample sizes. Our future directions will first aim to address these limitations, which would enable a more comprehensive joint effect analysis between the language style, individual traits, and preferences, and further investigate the potential causal relationship between and beyond these variables.

Figures (3)

• Figure 1: Methodological overview of Study 1 (Left) and Study 2 (Right). Experimental GUI for Study 2 is attached in Appendix \ref{['sec:app_gui']}.
• Figure 2: Example of Polarizing Moderator Effects
• Figure 3: User interface for our sampling with people experiment. During the actual experiment, the style names and intensity levels are not shown to the users, and the order of both are randomized for every iteration.