The Pitfalls of Growing Group Complexity: LLMs and Social Choice-Based Aggregation for Group Recommendations

TL;DR

This work investigates whether LLMs can correctly implement social choice-based aggregation for group recommendations under zero-shot and few-shot prompting. By testing 1,000 synthetic groups across 4 aggregation strategies and 4 LLMs, it demonstrates that group complexity degrades accuracy starting around $G = group extunderscore size \times num extunderscore items = 100$, but In-Context Learning markedly improves performance at higher complexity, while explanations or domain cues offer limited benefits. The study also shows that data formatting and model size influence results, with smaller open-source models capable of effective group recommendations when prompts are well-designed. Overall, it argues for including group complexity in GRS evaluations and highlights the potential of small, local LLMs to enable cost-efficient, privacy-friendly group recommendation pipelines.

Abstract

Large Language Models (LLMs) are increasingly applied in recommender systems aimed at both individuals and groups. Previously, Group Recommender Systems (GRS) often used social choice-based aggregation strategies to derive a single recommendation based on the preferences of multiple people. In this paper, we investigate under which conditions language models can perform these strategies correctly based on zero-shot learning and analyse whether the formatting of the group scenario in the prompt affects accuracy. We specifically focused on the impact of group complexity (number of users and items), different LLMs, different prompting conditions, including In-Context learning or generating explanations, and the formatting of group preferences. Our results show that performance starts to deteriorate when considering more than 100 ratings. However, not all language models were equally sensitive to growing group complexity. Additionally, we showed that In-Context Learning (ICL) can significantly increase the performance at higher degrees of group complexity, while adding other prompt modifications, specifying domain cues or prompting for explanations, did not impact accuracy. We conclude that future research should include group complexity as a factor in GRS evaluation due to its effect on LLM performance. Furthermore, we showed that formatting the group scenarios differently, such as rating lists per user or per item, affected accuracy. All in all, our study implies that smaller LLMs are capable of generating group recommendations under the right conditions, making the case for using smaller models that require less computing power and costs.

Figures (3)

• Figure 1: Pipeline for evaluating large language models' application of social choice-based aggregation strategies. We iterate through this pipeline for each given group scenario.
• Figure 2: Performance (Accuracy) of all LLMs (Llama, Mistral, Gemma and Phi4) based on 1,000 group scenarios with varying degrees of group complexity. Complexity ranges from 10 (2 members x 5 items) to 400 (8 members x 50 items).
• Figure 3: Performance of Phi4 based on prompt adjustments: (i) requiring explanations, (ii) implementing in-context learning and, (iii) adding domain cues. Group complexity was set at 100, 200 and 400 ratings (50 items; 2, 4 or 8 group members).