ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks

TL;DR

The paper addresses the challenge of evaluating retrieval-augmented generation in dynamic domains by introducing ChronoPlay, a framework that models dual dynamics—Knowledge Evolution and User Interest Drift—in automated benchmark generation. It combines a dual-source synthesis pipeline (authority knowledge plus authentic player patterns) with a dual-dynamic update mechanism (knowledge-driven and interest-driven updates) to produce time-evolving benchmarks ${\mathcal{B}}_t=(\mathcal{K}_t,\mathcal{D}_t)$. Through instantiations on three live-service games (DL2, Dune, PUBG Mobile), ChronoPlay reveals substantial lifecycle-based volatility in both retrieval and generation, demonstrates the necessity of modeling both dynamics for realistic evaluation, and shows that its synthesis modules (notably question templates and user personas) are critical for authentic benchmarks. The results underscore the importance of dynamic, authentic benchmarks for advancing robust, user-aligned RAG systems in gaming and potentially other evolving domains.

Abstract

Retrieval Augmented Generation (RAG) systems are increasingly vital in dynamic domains like online gaming, yet the lack of a dedicated benchmark has impeded standardized evaluation in this area. The core difficulty lies in Dual Dynamics: the constant interplay between game content updates and the shifting focus of the player community. Furthermore, the necessity of automating such a benchmark introduces a critical requirement for player-centric authenticity to ensure generated questions are realistic. To address this integrated challenge, we introduce ChronoPlay, a novel framework for the automated and continuous generation of game RAG benchmarks. ChronoPlay utilizes a dual-dynamic update mechanism to track both forms of change, and a dual-source synthesis engine that draws from official sources and player community to ensure both factual correctness and authentic query patterns. We instantiate our framework on three distinct games to create the first dynamic RAG benchmark for the gaming domain, offering new insights into model performance under these complex and realistic conditions. Code is avaliable at: https://github.com/hly1998/ChronoPlay.

Figures (15)

• Figure 1: An illustration of Dual Dynamics in Game, including Knowledge Evolution, which traces the game's knowledge updates, and User Interest Drift, which maps the changing interests of players.
• Figure 2: Distribution of main topics over time in Dying Light 2.
• Figure 3: An illustration of the Dual-Source Synthesis Pipeline. It leverages the Authority Knowledge Base $\mathcal{K}_{auth}$ for factual grounding, and uses the Question Template Base $\mathcal{T}_{comm}$ and the User Persona $\mathcal{U}_{comm}$ for authentic question patterns.
• Figure 4: An illustration of the Dual-Dynamic Update Mechanism, showing both Knowledge Evolution and User Interest Drift pathways.
• Figure 5: Generator correctness scores across the phases of each game.