CreAgent: Towards Long-Term Evaluation of Recommender System under Platform-Creator Information Asymmetry

TL;DR

CreAgent introduces an LLM-powered creator simulation agent designed to capture long-term RS dynamics under platform-creator information asymmetry. By integrating a belief module, memory, slow/fast thinking, and PPO-based fine-tuning, CreAgent behavior aligns with real-world creator patterns and enables robust evaluation of fairness- and diversity-aware RS strategies over time. The study demonstrates credible creator-Platform interactions, scalable simulation costs, and nuanced long-term impacts on user engagement, creator protection, and content enrichment. This framework offers a practical, cost-effective alternative to online A/B tests for multi-stakeholder RS evaluation and supports systematic exploration of long-term platform policies.

Abstract

Ensuring the long-term sustainability of recommender systems (RS) emerges as a crucial issue. Traditional offline evaluation methods for RS typically focus on immediate user feedback, such as clicks, but they often neglect the long-term impact of content creators. On real-world content platforms, creators can strategically produce and upload new items based on user feedback and preference trends. While previous studies have attempted to model creator behavior, they often overlook the role of information asymmetry. This asymmetry arises because creators primarily have access to feedback on the items they produce, while platforms possess data on the entire spectrum of user feedback. Current RS simulators, however, fail to account for this asymmetry, leading to inaccurate long-term evaluations. To address this gap, we propose CreAgent, a Large Language Model (LLM)-empowered creator simulation agent. By incorporating game theory's belief mechanism and the fast-and-slow thinking framework, CreAgent effectively simulates creator behavior under conditions of information asymmetry. Additionally, we enhance CreAgent's simulation ability by fine-tuning it using Proximal Policy Optimization (PPO). Our credibility validation experiments show that CreAgent aligns well with the behaviors between real-world platform and creator, thus improving the reliability of long-term RS evaluations. Moreover, through the simulation of RS involving CreAgents, we can explore how fairness- and diversity-aware RS algorithms contribute to better long-term performance for various stakeholders. CreAgent and the simulation platform are publicly available at https://github.com/shawnye2000/CreAgent.

Figures (10)

• Figure 1: (a) A platform where users favor movies, food, and sports in decreasing order, with a creator who has created sports and food items. A comparison of the creator's creation behavior under (b) limited information; and (c) full information hypothetically.
• Figure 2: The overall workflow of our simulation platform. (a) CreAgent, initialized with the real-world YouTube dataset, employs a belief mechanism combined with fast-and-slow thinking to create the next item strategically based on platform-provided user feedback. (b) Platform environment, which consists of an extensible two-stage recommendation system and modified widely-used user agent, collects item feedback from users and sends it to CreAgent.
• Figure 3: Comparison between the creation genre preference, diversity, and activity of the ground-truth and agent-simulated result.
• Figure 4: A comparison example between CreAgent and content generated by real-world YouTubers.
• Figure 5: An example of the impact of fine-tuning on improving the content generated by CreAgent.