Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ranking Heterogeneous Search Result Pages using the Interactive Probability Ranking Principle

Kanaad Pathak, Leif Azzopardi, Martin Halvey

TL;DR

This work extends the Probability Ranking Principle (PRP) to the Interactive Probability Ranking Principle (iPRP) within heterogeneous SERPs by incorporating interaction costs and encounter-based benefits through the Card Model. It defines the Expected Perceived Utility (EPU) for individual cards and lists, $EPU_{card}(i)$ and $EPU(\boldsymbol{L})$, under space constraints, using time-based measures for costs and benefits and BM25-derived relevance probabilities. Through a controlled crowdsourced study on the WaPo dataset with four card types, the authors estimate interaction probabilities and timings, illustrating that card presentation significantly modulates user behavior and the overall page utility. The results demonstrate notable shifts in ranking when optimizing by EPU rather than pure relevance, with measurable changes in RBO, DCG, and TBG, underscoring the need to account for presentation in SERP ranking and the potential for user-centric, adaptive layouts with practical implications and ethical considerations.

Abstract

The Probability Ranking Principle (PRP) ranks search results based on their expected utility derived solely from document contents, often overlooking the nuances of presentation and user interaction. However, with the evolution of Search Engine Result Pages (SERPs), now comprising a variety of result cards, the manner in which these results are presented is pivotal in influencing user engagement and satisfaction. This shift prompts the question: How does the PRP and its user-centric counterpart, the Interactive Probability Ranking Principle (iPRP), compare in the context of these heterogeneous SERPs? Our study draws a comparison between the PRP and the iPRP, revealing significant differences in their output. The iPRP, accounting for item-specific costs and interaction probabilities to determine the ``Expected Perceived Utility" (EPU), yields different result orderings compared to the PRP. We evaluate the effect of the EPU on the ordering of results by observing changes in the ranking within a heterogeneous SERP compared to the traditional ``ten blue links''. We find that changing the presentation affects the ranking of items according to the (iPRP) by up to 48\% (with respect to DCG, TBG and RBO) in ad-hoc search tasks on the TREC WaPo Collection. This work suggests that the iPRP should be employed when ranking heterogeneous SERPs to provide a user-centric ranking that adapts the ordering based on the presentation and user engagement.

Ranking Heterogeneous Search Result Pages using the Interactive Probability Ranking Principle

TL;DR

This work extends the Probability Ranking Principle (PRP) to the Interactive Probability Ranking Principle (iPRP) within heterogeneous SERPs by incorporating interaction costs and encounter-based benefits through the Card Model. It defines the Expected Perceived Utility (EPU) for individual cards and lists, and , under space constraints, using time-based measures for costs and benefits and BM25-derived relevance probabilities. Through a controlled crowdsourced study on the WaPo dataset with four card types, the authors estimate interaction probabilities and timings, illustrating that card presentation significantly modulates user behavior and the overall page utility. The results demonstrate notable shifts in ranking when optimizing by EPU rather than pure relevance, with measurable changes in RBO, DCG, and TBG, underscoring the need to account for presentation in SERP ranking and the potential for user-centric, adaptive layouts with practical implications and ethical considerations.

Abstract

The Probability Ranking Principle (PRP) ranks search results based on their expected utility derived solely from document contents, often overlooking the nuances of presentation and user interaction. However, with the evolution of Search Engine Result Pages (SERPs), now comprising a variety of result cards, the manner in which these results are presented is pivotal in influencing user engagement and satisfaction. This shift prompts the question: How does the PRP and its user-centric counterpart, the Interactive Probability Ranking Principle (iPRP), compare in the context of these heterogeneous SERPs? Our study draws a comparison between the PRP and the iPRP, revealing significant differences in their output. The iPRP, accounting for item-specific costs and interaction probabilities to determine the ``Expected Perceived Utility" (EPU), yields different result orderings compared to the PRP. We evaluate the effect of the EPU on the ordering of results by observing changes in the ranking within a heterogeneous SERP compared to the traditional ``ten blue links''. We find that changing the presentation affects the ranking of items according to the (iPRP) by up to 48\% (with respect to DCG, TBG and RBO) in ad-hoc search tasks on the TREC WaPo Collection. This work suggests that the iPRP should be employed when ranking heterogeneous SERPs to provide a user-centric ranking that adapts the ordering based on the presentation and user engagement.
Paper Structure (8 sections, 5 equations, 5 figures, 2 tables)

This paper contains 8 sections, 5 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Proposed Method
  4. Experimental Methodology
  5. Estimating the EPU
  6. Results
  7. Discussion & Future Work
  8. Acknowledgements

Figures (5)

  • Figure 1: Compared to SERP A, only four cards can be shown above the fold (dotted horizontal blue line) on SERP B and C. However, changing the card type (e.g., TS to TIS) may also lead to changes in the ranking under the iPRP.
  • Figure 2: Title+Image+Summary (TIS)
  • Figure 3: Title Only (T)
  • Figure 4: Title+Summary (TS)
  • Figure 5: Title+Image (TI)