Examining the Role of Relationship Alignment in Large Language Models

TL;DR

The ability of Llama 3.0 (70B) to predict the semantic tones across different combinations of a commenter's and poster's gender, age, and friendship closeness is evaluated and it is demonstrated that LLMs can comprehend semantics from the original post alone.

Abstract

The rapid development and deployment of Generative AI in social settings raise important questions about how to optimally personalize them for users while maintaining accuracy and realism. Based on a Facebook public post-comment dataset, this study evaluates the ability of Llama 3.0 (70B) to predict the semantic tones across different combinations of a commenter's and poster's gender, age, and friendship closeness and to replicate these differences in LLM-generated comments. The study consists of two parts: Part I assesses differences in semantic tones across social relationship categories, and Part II examines the similarity between comments generated by Llama 3.0 (70B) and human comments from Part I given public Facebook posts as input. Part I results show that including social relationship information improves the ability of a model to predict the semantic tone of human comments. However, Part II results show that even without including social context information in the prompt, LLM-generated comments and human comments are equally sensitive to social context, suggesting that LLMs can comprehend semantics from the original post alone. When we include all social relationship information in the prompt, the similarity between human comments and LLM-generated comments decreases. This inconsistency may occur because LLMs did not include social context information as part of their training data. Together these results demonstrate the ability of LLMs to comprehend semantics from the original post and respond similarly to human comments, but also highlights their limitations in generalizing personalized comments through prompting alone.

Figures (6)

• Figure 1: Example of LLM prompt without and with social context information, paraphrased human comment, and LLM generated comments from prompts without and with social context information.
• Figure 2: Multi-level regression results from Part I predicting semantic tone outcomes from social context variables and post topic (not shown). Post ID is included as a random effect, since multiple comments can be nested under a single post. The conditional $R^{2}$ represents the variation explained by both fixed and random effects, while marginal $R^{2}$ represents the variation explained by fixed effects only. *** $p<0.001$.
• Figure 3: Mixed effect regressions predicting the log-transformed sentiment scores of LLM generated comments from the tone of human comments with and without social context information. The dashed line indicates perfect prediction, where the tone of generated comments is perfectly predicted from the tone of human comments. The gray line shows fitted results for LLM comments without social context, and the blue line shows fitted results for LLM comments with social context. The left panel with results for the “insult score” shows that including social context in the prompt slightly increased the similarity between the human and generated comments. In contrast, the right panel with results for the “self-disclosure score” shows that including the context decreases the similarity.
• Figure 4: Regression coefficients of social context factors in the fitted models for log-transformed insult score (left panel) and self-disclosure score (right) based on Approach 2. We compare human comments (red), generate comments when provided social context (blue), and generated comments without provided social context (gray). The horizontal bar shows 95% confidence interval. Plots for other semantic tones are shown in Figure \ref{['fig-3-appendix']} in the Appendix.
• Figure 5: We describe the semantic categories used in this paper. For the post and comment examples, we paraphrase them.