RAIR: A Rule-Aware Benchmark Uniting Challenging Long-Tail and Visual Salience Subset for E-commerce Relevance Assessment

TL;DR

RAIR addresses the need for a rigorous, rule-grounded relevance benchmark in e-commerce by introducing a Rule-Aware benchmark with images. It formalizes relevance as a four-level, attribute-rich task and augments judgments with a standardized set of rules, enabling objective evaluation across knowledge, reasoning, and multimodal understanding. The dataset comprises 63,601 samples across general, long-tail hard, and visual salience subsets, with a dedicated rule checklist for evaluation and three industry-balanced generalizations. Empirical results show RAIR is sufficiently challenging, distinguishing model capabilities, and reveal that rule-guided reasoning and multimodal data substantially affect performance, especially for large, rule-aware LLMs and VLMs; the benchmark thus provides a practical industry-standard for evaluating contemporary relevance models and guiding future improvements.

Abstract

Search relevance plays a central role in web e-commerce. While large language models (LLMs) have shown significant results on relevance task, existing benchmarks lack sufficient complexity for comprehensive model assessment, resulting in an absence of standardized relevance evaluation metrics across the industry. To address this limitation, we propose Rule-Aware benchmark with Image for Relevance assessment(RAIR), a Chinese dataset derived from real-world scenarios. RAIR established a standardized framework for relevance assessment and provides a set of universal rules, which forms the foundation for standardized evaluation. Additionally, RAIR analyzes essential capabilities required for current relevance models and introduces a comprehensive dataset consists of three subset: (1) a general subset with industry-balanced sampling to evaluate fundamental model competencies; (2) a long-tail hard subset focus on challenging cases to assess performance limits; (3) a visual salience subset for evaluating multimodal understanding capabilities. We conducted experiments on RAIR using 14 open and closed-source models. The results demonstrate that RAIR presents sufficient challenges even for GPT-5, which achieved the best performance. RAIR data are now available, serving as an industry benchmark for relevance assessment while providing new insights into general LLM and Visual Language Model(VLM) evaluation.

Figures (6)

• Figure 1: Overview of three core model capabilities evaluated in the RAIR benchmark.
• Figure 2: The Relevance Rule Framework of RAIR
• Figure 3: Construction process of RAIR benchmark, including the rule system composition and construction pipelines for general subset, long-tail subset, and visual salience subset.
• Figure 4: Distribution of error categories for the Qwen3-235B-a22B-Instruct-2507 model on the general set and the hard subset.
• Figure 5: Industry hard case