LLMs Homogenize Values in Constructive Arguments on Value-Laden Topics

TL;DR

This study investigates how large language models reshape underlying human values when rewriting comments on value-laden topics. Through a three-phase cross-cultural experiment with 465 participants from India and the United States, the authors compare human-written versus GPT-4 rewritten constructive comments on homophobic and Islamophobic threads and analyze value framings via Schwartz's framework. They find that LLMs systematically dampen Conservative values while boosting prosocial values like Benevolence and Universalism, sometimes shifting authors' stances toward neutrality or support. Perceived alignment varies by reader stance, with conservatives tending to favor human-written comments and liberals favoring LLM rewrites, highlighting potential marginalization of certain viewpoints and pressing design considerations for online discourse platforms.

Abstract

Large language models (LLMs) are increasingly used to promote prosocial and constructive discourse online. Yet little is known about how these models negotiate and shape underlying values when reframing people's arguments on value-laden topics. We conducted experiments with 465 participants from India and the United States, who wrote comments on homophobic and Islamophobic threads, and reviewed human-written and LLM-rewritten constructive versions of these comments. Our analysis shows that LLM systematically diminishes Conservative values while elevating prosocial values such as Benevolence and Universalism. When these comments were read by others, participants opposing same-sex marriage or Islam found human-written comments more aligned with their values, whereas those supportive of these communities found LLM-rewritten versions more aligned with their values. These findings suggest that value homogenization in LLM-mediated prosocial discourse runs the risk of marginalizing conservative viewpoints on value-laden topics and may inadvertently shape the dynamics of online discourse.

Figures (7)

• Figure 1: An outline of our data collection methods in each phase and key findings.
• Figure 2: Circular motivational continuum of Schwartz's basic human values (adapted from schwartz2012refining).
• Figure 3: Annotated Schwartz's human values (blue texts) in a comment written by an Indian participant on the homophobic thread.
• Figure 4: The interface for human-LLM collaborative writing of constructive comments. Participants first entered their comment in the textbox for User Comment and then selected one or more prompts from Instructions for AI section. After they clicked “Make Comment Constructive”, GPT-4 rewrote their comment constructively based on given instructions in real time, which appeared in the AI Suggested Comment box. Participants could repeat the process as many times as they wished. The example shown illustrates a comment written by an American participant in response to an Islamophobic thread.
• Figure 5: Distribution of Schwartz's human values in (A) all comments written by Indian and American participants, (B) comments written on homophobic thread by participants who either support or oppose same-sex marriage. Statistically significant differences are reported at $p<0.00001$ (****), $p<0.0001$ (***), $p<0.001$ (**), and $p<0.01$ (*)[adjusted P-values after Bonferroni correction].