Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Contrastive Feedback for Effective User Simulations

Andreas Konstantin Kruff, Timo Breuer, Philipp Schaer

TL;DR

This paper addresses how prompting strategies influence LLM-based user simulations in Interactive Information Retrieval by testing contrastive prompting where summaries of seen relevant or irrelevant documents guide the model’s knowledge state. The authors implement eight user configurations and compare prompting modalities (topic-based context vs. summaries) within the SimIIR3 framework using BM25 and a single Llama3.3 model to generate queries and judgments, evaluating with information gain ($IG$) and session-discounted Cumulative Gain ($sDCG$). They find that contrastive prompting generally improves simulation effectiveness, with CRF frequently performing best (notably on Core17) and PRF leading on Core18, while some topic-only prompts can drift without full topic context. The study highlights practical benefits for generating realistic synthetic interaction data and points to limitations related to pooling-reported relevance judgments and the need for richer evaluation resources for unjudged documents. Overall, the results support adopting contrastive prompts to enhance user simulations in IR, informing future work on more robust evaluation frameworks and unjudged-content handling.

Abstract

The use of Large Language Models (LLMs) for simulating user behavior in the domain of Interactive Information Retrieval has recently gained significant popularity. However, their application and capabilities remain highly debated and understudied. This study explores whether the underlying principles of contrastive training techniques, which have been effective for fine-tuning LLMs, can also be applied beneficially in the area of prompt engineering for user simulations. Previous research has shown that LLMs possess comprehensive world knowledge, which can be leveraged to provide accurate estimates of relevant documents. This study attempts to simulate a knowledge state by enhancing the model with additional implicit contextual information gained during the simulation. This approach enables the model to refine the scope of desired documents further. The primary objective of this study is to analyze how different modalities of contextual information influence the effectiveness of user simulations. Various user configurations were tested, where models are provided with summaries of already judged relevant, irrelevant, or both types of documents in a contrastive manner. The focus of this study is the assessment of the impact of the prompting techniques on the simulated user agent performance. We hereby lay the foundations for leveraging LLMs as part of more realistic simulated users.

Evaluating Contrastive Feedback for Effective User Simulations

TL;DR

This paper addresses how prompting strategies influence LLM-based user simulations in Interactive Information Retrieval by testing contrastive prompting where summaries of seen relevant or irrelevant documents guide the model’s knowledge state. The authors implement eight user configurations and compare prompting modalities (topic-based context vs. summaries) within the SimIIR3 framework using BM25 and a single Llama3.3 model to generate queries and judgments, evaluating with information gain () and session-discounted Cumulative Gain (). They find that contrastive prompting generally improves simulation effectiveness, with CRF frequently performing best (notably on Core17) and PRF leading on Core18, while some topic-only prompts can drift without full topic context. The study highlights practical benefits for generating realistic synthetic interaction data and points to limitations related to pooling-reported relevance judgments and the need for richer evaluation resources for unjudged documents. Overall, the results support adopting contrastive prompts to enhance user simulations in IR, informing future work on more robust evaluation frameworks and unjudged-content handling.

Abstract

The use of Large Language Models (LLMs) for simulating user behavior in the domain of Interactive Information Retrieval has recently gained significant popularity. However, their application and capabilities remain highly debated and understudied. This study explores whether the underlying principles of contrastive training techniques, which have been effective for fine-tuning LLMs, can also be applied beneficially in the area of prompt engineering for user simulations. Previous research has shown that LLMs possess comprehensive world knowledge, which can be leveraged to provide accurate estimates of relevant documents. This study attempts to simulate a knowledge state by enhancing the model with additional implicit contextual information gained during the simulation. This approach enables the model to refine the scope of desired documents further. The primary objective of this study is to analyze how different modalities of contextual information influence the effectiveness of user simulations. Various user configurations were tested, where models are provided with summaries of already judged relevant, irrelevant, or both types of documents in a contrastive manner. The focus of this study is the assessment of the impact of the prompting techniques on the simulated user agent performance. We hereby lay the foundations for leveraging LLMs as part of more realistic simulated users.
Paper Structure (9 sections, 1 equation, 2 figures)

This paper contains 9 sections, 1 equation, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. User Configurations
  5. Prompting Techniques
  6. Implementation Details and Datasets
  7. Evaluation Measures
  8. Experimental Results
  9. Conclusion and Future Work

Figures (2)

  • Figure 1: A flowchart of the analyzed user configuration described in Section \ref{['subsec:user_config']}. The color codes correspond to those in Figure \ref{['fig:experimental_results']} that show the experimental results in the simulated search sessions.
  • Figure 2: Results of the simulated sessions for different prompting strategies evaluated with the help of Effort-Effect trade-offs and sDCG on Core17 (top) and Core18 (bottom).