CSP: A Simulator For Multi-Agent Ranking Competitions

TL;DR

The paper addresses how ranking incentives shape corpus dynamics when document authors are LLM-based agents. It introduces the CSP simulator, along with CSPAnalyzer and CSPCompare, to run large-scale, configurable ranking competitions and analyze resultant datasets. Through 22 LLM-driven simulations and comparisons with a human-compiled dataset, it finds that LLMs tend to mimic top-ranked content and reduce diversity more than humans, with the language model playing a larger role in dynamics than the ranking function. This scalable platform enables systematic study of competitive search and AI-generated web content, offering insights for understanding potential shifts in web ecosystems and informing ranking system designs. The work provides public code and data to support further research in this area.

Abstract

In ranking competitions, document authors compete for the highest rankings by modifying their content in response to past rankings. Previous studies focused on human participants, primarily students, in controlled settings. The rise of generative AI, particularly Large Language Models (LLMs), introduces a new paradigm: using LLMs as document authors. This approach addresses scalability constraints in human-based competitions and reflects the growing role of LLM-generated content on the web-a prime example of ranking competition. We introduce a highly configurable ranking competition simulator that leverages LLMs as document authors. It includes analytical tools to examine the resulting datasets. We demonstrate its capabilities by generating multiple datasets and conducting an extensive analysis. Our code and datasets are publicly available for research.

Figures (19)

• Figure 1: Average absolute difference of feature values of winner documents in rounds $i$ ($W_i$) and $i+1$ ($W_{i+1}$).
• Figure 2: Class: M.
• Figure 3: Class: M.
• Figure 4: Class: D, C.
• Figure 5: Class: D,C.