Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)

TL;DR

Evaluating information access (IA) systems is challenging due to complex user interactions and limitations of Cranfield-style experiments. This paper outlines the second Sim4IA workshop at SIGIR 2025, focused on user simulations and two micro shared tasks implemented with the SimIIR framework to enable reproducible, realistic evaluations and community building. Task A simulates interactions with ranked lists using a LongEval 2025 CLEF-derived dataset (queries, SERPs, clicks), while Task B simulates next-user utterances in conversational IA, evaluated by semantic similarity to real utterances; both tasks are designed with dockerized baselines and held-out data. The workshop aims to foster cross-disciplinary dialogue, advertise a potential TREC/CLEF campaign, and produce practical artifacts (code, prompts, lab notes) to advance simulation-based evaluation in IA.

Abstract

Simulations in information access (IA) have recently gained interest, as shown by various tutorials and workshops around that topic. Simulations can be key contributors to central IA research and evaluation questions, especially around interactive settings when real users are unavailable, or their participation is impossible due to ethical reasons. In addition, simulations in IA can help contribute to a better understanding of users, reduce complexity of evaluation experiments, and improve reproducibility. Building on recent developments in methods and toolkits, the second iteration of our Sim4IA workshop aims to again bring together researchers and practitioners to form an interactive and engaging forum for discussions on the future perspectives of the field. An additional aim is to plan an upcoming TREC/CLEF campaign.

Figures (1)

• Figure 1: Shared task design that provides participants with prepared development environments that help defining user simulation configurations, which will be evaluated by the organizers with held-out data and systems.