How role-play shapes relevance judgment in zero-shot LLM rankers

TL;DR

This paper investigates how role-play prompts shape zero-shot LLM rankers and why prompting is impactful. It combines a modular role-play template with activation patching, a mechanistic interpretability technique, to trace how role information propagates through Transformer layers and influences relevance judgments. The study identifies that role signals originate in early layers, interact modestly with instruction tokens in middle layers, and largely ignore query/document representations, with certain attention heads causally mediating these effects. It demonstrates the robustness of these findings across multiple models and datasets, and provides concrete guidance for designing more effective role-play prompts in IR and other zero-shot tasks. The work offers both theoretical insight into internal model mechanisms and practical prompt-design prescriptions for real-world retrieval systems.

Abstract

Large Language Models (LLMs) have emerged as promising zero-shot rankers, but their performance is highly sensitive to prompt formulation. In particular, role-play prompts, where the model is assigned a functional role or identity, often give more robust and accurate relevance rankings. However, the mechanisms and diversity of role-play effects remain underexplored, limiting both effective use and interpretability. In this work, we systematically examine how role-play variations influence zero-shot LLM rankers. We employ causal intervention techniques from mechanistic interpretability to trace how role-play information shapes relevance judgments in LLMs. Our analysis reveals that (1) careful formulation of role descriptions have a large effect on the ranking quality of the LLM; (2) role-play signals are predominantly encoded in early layers and communicate with task instructions in middle layers, while receiving limited interaction with query or document representations. Specifically, we identify a group of attention heads that encode information critical for role-conditioned relevance. These findings not only shed light on the inner workings of role-play in LLM ranking but also offer guidance for designing more effective prompts in IR and beyond, pointing toward broader opportunities for leveraging role-play in zero-shot applications.

Figures (9)

• Figure 1: Role-play activation patching for zero-shot rankings.
• Figure 2: Zero-shot LLM ranker performance (nDCG@10) for various role-play wordings across methods, prompt types, and datasets on LLaMA-3.1-8B-Instruct. Baselines without role-play appear as horizontal lines.
• Figure 3: Results of residual stream patching at layer inputs across prompt segment positions on LLaMa-3.1-8B-Instruct. Darker blue reflects stronger gains.
• Figure 4: Patching result of different segment orders for the pointwise ranking.
• Figure 5: Patching results of attention heads on role, instruction and last token for pointwise ranking.