Report on the Workshop on Simulations for Information Access (Sim4IA 2024) at SIGIR 2024

TL;DR

The paper reports on the Sim4IA 2024 workshop at SIGIR 2024, documenting its organization, schedule, and a synthesis of discussions around simulations for information access. It presents two keynotes, a panel, nine lightning talks, and two breakout groups, focusing on the role of user simulations to evaluate information access technologies and to bridge offline and online experiments. The contributions include perspectives on personalization, model-based evaluation, and shared-task design, highlighting open challenges in model fidelity, applicability, and data requirements. The findings underscore significant interest across academia and industry in advancing simulation-based evaluation and in coordinating a community effort for shared tasks. The work lays groundwork for future events and methodological frameworks, with RecSim NG as a notable tooling reference.

Abstract

This paper is a report of the Workshop on Simulations for Information Access (Sim4IA) workshop at SIGIR 2024. The workshop had two keynotes, a panel discussion, nine lightning talks, and two breakout sessions. Key takeaways were user simulation's importance in academia and industry, the possible bridging of online and offline evaluation, and the issues of organizing a companion shared task around user simulations for information access. We report on how we organized the workshop, provide a brief overview of what happened at the workshop, and summarize the main topics and findings of the workshop and future work.

Figures (2)

• Figure 1: Left: Panel discussion. Right: Breakout group participants.
• Figure 2: Depiction of the multi-layered factors of a user for the example task of learning about evaluation methodologies for RAG.