Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sim4IA-Bench: A User Simulation Benchmark Suite for Next Query and Utterance Prediction

Andreas Konstantin Kruff, Christin Katharina Kreutz, Timo Breuer, Philipp Schaer, Krisztian Balog

TL;DR

Sim4IA-Bench addresses the lack of standardized evaluation resources for user simulators in information retrieval by providing a public benchmark that directly links real user session logs to simulated next-query and next-utterance outputs. It builds on the Sim4IA Micro-Shared Task (Task A and B) and provides datasets, submission runs, a baseline toolkit, and new similarity-based measures to assess simulator fidelity, independent of downstream retrieval performance. The contributions include a two-task dataset derived from CORE logs, a suite of evaluation measures (semantic similarity, redundancy, SERP overlap, rank-diversity), and a reusable toolkit to enable reproducible research. Together, these resources enable rigorous benchmarking of simulators, support studies of query reformulation and intent drift, and lay the groundwork for future community-driven evaluation campaigns in interactive IR and conversational search.

Abstract

Validating user simulation is a difficult task due to the lack of established measures and benchmarks, which makes it challenging to assess whether a simulator accurately reflects real user behavior. As part of the Sim4IA Micro-Shared Task at the Sim4IA Workshop, SIGIR 2025, we present Sim4IA-Bench, a simulation benchmark suit for the prediction of the next queries and utterances, the first of its kind in the IR community. Our dataset as part of the suite comprises 160 real-world search sessions from the CORE search engine. For 70 of these sessions, up to 62 simulator runs are available, divided into Task A and Task B, in which different approaches predicted users next search queries or utterances. Sim4IA-Bench provides a basis for evaluating and comparing user simulation approaches and for developing new measures of simulator validity. Although modest in size, the suite represents the first publicly available benchmark that links real search sessions with simulated next-query predictions. In addition to serving as a testbed for next query prediction, it also enables exploratory studies on query reformulation behavior, intent drift, and interaction-aware retrieval evaluation. We also introduce a new measure for evaluating next-query predictions in this task. By making the suite publicly available, we aim to promote reproducible research and stimulate further work on realistic and explainable user simulation for information access: https://github.com/irgroup/Sim4IA-Bench.

Sim4IA-Bench: A User Simulation Benchmark Suite for Next Query and Utterance Prediction

TL;DR

Sim4IA-Bench addresses the lack of standardized evaluation resources for user simulators in information retrieval by providing a public benchmark that directly links real user session logs to simulated next-query and next-utterance outputs. It builds on the Sim4IA Micro-Shared Task (Task A and B) and provides datasets, submission runs, a baseline toolkit, and new similarity-based measures to assess simulator fidelity, independent of downstream retrieval performance. The contributions include a two-task dataset derived from CORE logs, a suite of evaluation measures (semantic similarity, redundancy, SERP overlap, rank-diversity), and a reusable toolkit to enable reproducible research. Together, these resources enable rigorous benchmarking of simulators, support studies of query reformulation and intent drift, and lay the groundwork for future community-driven evaluation campaigns in interactive IR and conversational search.

Abstract

Validating user simulation is a difficult task due to the lack of established measures and benchmarks, which makes it challenging to assess whether a simulator accurately reflects real user behavior. As part of the Sim4IA Micro-Shared Task at the Sim4IA Workshop, SIGIR 2025, we present Sim4IA-Bench, a simulation benchmark suit for the prediction of the next queries and utterances, the first of its kind in the IR community. Our dataset as part of the suite comprises 160 real-world search sessions from the CORE search engine. For 70 of these sessions, up to 62 simulator runs are available, divided into Task A and Task B, in which different approaches predicted users next search queries or utterances. Sim4IA-Bench provides a basis for evaluating and comparing user simulation approaches and for developing new measures of simulator validity. Although modest in size, the suite represents the first publicly available benchmark that links real search sessions with simulated next-query predictions. In addition to serving as a testbed for next query prediction, it also enables exploratory studies on query reformulation behavior, intent drift, and interaction-aware retrieval evaluation. We also introduce a new measure for evaluating next-query predictions in this task. By making the suite publicly available, we aim to promote reproducible research and stimulate further work on realistic and explainable user simulation for information access: https://github.com/irgroup/Sim4IA-Bench.

Paper Structure

This paper contains 19 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction and Motivation
  2. The Sim4IA-Bench Suite
  3. Related Work
  4. Validating User Simulators
  5. Datasets
  6. Dataset
  7. Task Descriptions and Dataset Contents
  8. Task A: Interactive IR Simulation
  9. Task B: Conversational Session Simulation
  10. Dataset Statistics
  11. Session Extraction from CORE Logs
  12. Usage Notes
  13. Submitted Runs
  14. Assessing Simulator Fidelity
  15. Measures
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Overview of Task A (interactive IR simulation, in orange) and Task B (conversational session simulation, in teal). Document IDs highlighted in green represent documents a real user interacted with.
  • Figure 2: Prompt template for generating the synthetical responses of the conversational system for Task B. Black parts show the unchanging structural and instructional parts, purple components depict the variables inserted depending on sessions and utterances.
  • Figure 3: Exemplary run results for Task A1.