Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Diagnostic Prompting Approach for Multimodal LLM-based Visual Complexity Assessment: A Case Study of Amazon Search Result Pages

Divendar Murtadak, Yoon Kim, Trilokya Akula

TL;DR

The paper tackles the challenge of reliably assessing visual complexity on Amazon search result pages at scale by testing diagnostic prompting for multimodal LLMs against standard Gestalt-based prompts, using 200 SRPs with human-ground truth. It demonstrates a sizable relative improvement in alignment and detection performance ($F1$ up to $0.297$, $\kappa$ up to $0.071$), while revealing persistent gaps in perception and domain generalization. The authors analyze model reasoning through decision trees and highlight divergences between human and MLLM drivers, notably badge clutter versus content similarity. They propose a concrete roadmap for enhancing reliability, including richer response scales, persona-specific prompting, and larger-scale validation beyond a single domain.

Abstract

This study investigates whether diagnostic prompting can improve Multimodal Large Language Model (MLLM) reliability for visual complexity assessment of Amazon Search Results Pages (SRP). We compare diagnostic prompting with standard gestalt principles-based prompting using 200 Amazon SRP pages and human expert annotations. Diagnostic prompting showed notable improvements in predicting human complexity judgments, with F1-score increasing from 0.031 to 0.297 (+858\% relative improvement), though absolute performance remains modest (Cohen's $κ$ = 0.071). The decision tree revealed that models prioritize visual design elements (badge clutter: 38.6\% importance) while humans emphasize content similarity, suggesting partial alignment in reasoning patterns. Failure case analysis reveals persistent challenges in MLLM visual perception, particularly for product similarity and color intensity assessment. Our findings indicate that diagnostic prompting represents a promising initial step toward human-aligned MLLM-based evaluation, though failure cases with consistent human-MLLM disagreement require continued research and refinement in prompting approaches with larger ground truth datasets for reliable practical deployment.

Exploring Diagnostic Prompting Approach for Multimodal LLM-based Visual Complexity Assessment: A Case Study of Amazon Search Result Pages

TL;DR

The paper tackles the challenge of reliably assessing visual complexity on Amazon search result pages at scale by testing diagnostic prompting for multimodal LLMs against standard Gestalt-based prompts, using 200 SRPs with human-ground truth. It demonstrates a sizable relative improvement in alignment and detection performance ( up to , up to ), while revealing persistent gaps in perception and domain generalization. The authors analyze model reasoning through decision trees and highlight divergences between human and MLLM drivers, notably badge clutter versus content similarity. They propose a concrete roadmap for enhancing reliability, including richer response scales, persona-specific prompting, and larger-scale validation beyond a single domain.

Abstract

This study investigates whether diagnostic prompting can improve Multimodal Large Language Model (MLLM) reliability for visual complexity assessment of Amazon Search Results Pages (SRP). We compare diagnostic prompting with standard gestalt principles-based prompting using 200 Amazon SRP pages and human expert annotations. Diagnostic prompting showed notable improvements in predicting human complexity judgments, with F1-score increasing from 0.031 to 0.297 (+858\% relative improvement), though absolute performance remains modest (Cohen's = 0.071). The decision tree revealed that models prioritize visual design elements (badge clutter: 38.6\% importance) while humans emphasize content similarity, suggesting partial alignment in reasoning patterns. Failure case analysis reveals persistent challenges in MLLM visual perception, particularly for product similarity and color intensity assessment. Our findings indicate that diagnostic prompting represents a promising initial step toward human-aligned MLLM-based evaluation, though failure cases with consistent human-MLLM disagreement require continued research and refinement in prompting approaches with larger ground truth datasets for reliable practical deployment.

Paper Structure

This paper contains 26 sections, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Customer Problem
  3. Related Work
  4. Data and Methodology
  5. Data
  6. Experimental Design
  7. Implementation
  8. Results
  9. Statistical Validation and Performance Analysis
  10. Model Interpretability and Feature Analysis
  11. Qualitative Validation Through Failure Cases
  12. Discussion
  13. Preliminary Findings and Framework Assessment
  14. Critical Limitations and Alignment Gaps
  15. Future Work
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Visual complexity challenges in Amazon search interfaces showing (left) similarity-based complexity where products appear nearly identical, (center) content overload with excessive information density, and (right) visual overload with competing visual elements.
  • Figure : Case 1: "Magic Eraser"
  • Figure : Case 1: "Magic Eraser"
  • Figure : Case 2: "Travel Perfume Bottle"