Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Influence of Presentation and Performance on User Satisfaction

Kanaad Pathak, Leif Azzopardi, Martin Halvey

TL;DR

This work addresses how query performance and result presentation jointly influence user satisfaction in information retrieval. Using a between-subjects design with five result-card layouts on the WaPo 2018 corpus, the study measures query-level satisfaction, interface-level satisfaction, and both system-side ($nDCG@10$) and user-side performance metrics, including Total Gain on the first page. It finds a strong linear relationship between $nDCG@10$ and query satisfaction, while presentation alone does not linearly elevate satisfaction; however, presentation interacts with total page gain to affect satisfaction and reveals a complex, multi-factorial pattern through LDA analysis. The results underscore the need to harmonize retrieval quality and result presentation in IR design, suggesting adaptive interfaces that account for diverse user preferences and task contexts.

Abstract

The effectiveness of an IR system is gauged not just by its ability to retrieve relevant results but also by how it presents these results to users; an engaging presentation often correlates with increased user satisfaction. While existing research has delved into the link between user satisfaction, IR performance metrics, and presentation, these aspects have typically been investigated in isolation. Our research aims to bridge this gap by examining the relationship between query performance, presentation and user satisfaction. For our analysis, we conducted a between-subjects experiment comparing the effectiveness of various result card layouts for an ad-hoc news search interface. Drawing data from the TREC WaPo 2018 collection, we centered our study on four specific topics. Within each of these topics, we assessed six distinct queries with varying nDCG values. Our study involved 164 participants who were exposed to one of five distinct layouts containing result cards, such as "title'', "title+image'', or "title+image+summary''. Our findings indicate that while nDCG is a strong predictor of user satisfaction at the query level, there exists no linear relationship between the performance of the query, presentation of results and user satisfaction. However, when considering the total gain on the initial result page, we observed that presentation does play a significant role in user satisfaction (at the query level) for certain layouts with result cards such as, title+image or title+image+summary. Our results also suggest that the layout differences have complex and multifaceted impacts on satisfaction. We demonstrate the capacity to equalize user satisfaction levels between queries of varying performance by changing how results are presented. This emphasizes the necessity to harmonize both performance and presentation in IR systems, considering users' diverse preferences.

The Influence of Presentation and Performance on User Satisfaction

TL;DR

This work addresses how query performance and result presentation jointly influence user satisfaction in information retrieval. Using a between-subjects design with five result-card layouts on the WaPo 2018 corpus, the study measures query-level satisfaction, interface-level satisfaction, and both system-side () and user-side performance metrics, including Total Gain on the first page. It finds a strong linear relationship between and query satisfaction, while presentation alone does not linearly elevate satisfaction; however, presentation interacts with total page gain to affect satisfaction and reveals a complex, multi-factorial pattern through LDA analysis. The results underscore the need to harmonize retrieval quality and result presentation in IR design, suggesting adaptive interfaces that account for diverse user preferences and task contexts.

Abstract

The effectiveness of an IR system is gauged not just by its ability to retrieve relevant results but also by how it presents these results to users; an engaging presentation often correlates with increased user satisfaction. While existing research has delved into the link between user satisfaction, IR performance metrics, and presentation, these aspects have typically been investigated in isolation. Our research aims to bridge this gap by examining the relationship between query performance, presentation and user satisfaction. For our analysis, we conducted a between-subjects experiment comparing the effectiveness of various result card layouts for an ad-hoc news search interface. Drawing data from the TREC WaPo 2018 collection, we centered our study on four specific topics. Within each of these topics, we assessed six distinct queries with varying nDCG values. Our study involved 164 participants who were exposed to one of five distinct layouts containing result cards, such as "title'', "title+image'', or "title+image+summary''. Our findings indicate that while nDCG is a strong predictor of user satisfaction at the query level, there exists no linear relationship between the performance of the query, presentation of results and user satisfaction. However, when considering the total gain on the initial result page, we observed that presentation does play a significant role in user satisfaction (at the query level) for certain layouts with result cards such as, title+image or title+image+summary. Our results also suggest that the layout differences have complex and multifaceted impacts on satisfaction. We demonstrate the capacity to equalize user satisfaction levels between queries of varying performance by changing how results are presented. This emphasizes the necessity to harmonize both performance and presentation in IR systems, considering users' diverse preferences.
Paper Structure (14 sections, 1 equation, 5 figures, 5 tables)

This paper contains 14 sections, 1 equation, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Collection and System
  5. Search Topics and Tasks
  6. Measures
  7. Procedure
  8. Participant Demographics
  9. Ethics Approval
  10. Experimental Results
  11. Summary of Search Behaviours
  12. RQ 1: How do the quality of search results (as measured by query performance) and the interface layout impact user satisfaction in information retrieval tasks?
  13. RQ 2: What are the effects of different interface layouts on user satisfaction as measured by overall satisfaction, the likability of the engine, productivity, and mental effort?
  14. Conclusion & Future Work

Figures (5)

  • Figure 1: Example of the different result card types, with an approximation of the number of rows each card type occupies.
  • Figure 2: An example of the user interface presented to participants for collection of annotations. Sub-figure (b) shows an example of a SERP layout with a random arrangement of cards.
  • Figure 3: The relationship between query satisfaction and nDCG@10
  • Figure 4: The relationship between query satisfaction and Total Gain on Page 1
  • Figure 5: Visualization of the first two Linear Discriminants (LD1 and LD2), for different interface layouts