HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application

TL;DR

HSCodeComp targets the crucial but largely overlooked ability of deep search agents to apply hierarchical, expert-level rules in a real-world domain. By framing HSCode prediction as a multi-hop, rule-guided task and constructing a 632-sample dataset with expert annotations, the authors evaluate a broad spectrum of LLM/VLM models and agent systems, revealing a substantial gap to human experts (best $46.8\%$ vs $95.0\%$). The study demonstrates that current approaches struggle with rule boundaries, multi-hop reasoning, and integration of structured tariff rules, and that simple tool use and multimodal inputs offer limited relief, while test-time scaling offers little benefit. The work provides a practical benchmark, dataset, and analysis that will catalyze development of more capable rule-aware deep search agents for e-commerce and beyond. The HSCodeComp framework, including its emphasis on hierarchical rule utilization and expert validation, has significant implications for real-world classification tasks where domain-specific regulations govern decision-making. $\text{HSCodeComp}$ thus serves as a benchmark for advancing robust, rule-grounded AI systems in complex, regulated environments.

Abstract

Effective deep search agents must not only access open-domain and domain-specific knowledge but also apply complex rules-such as legal clauses, medical manuals and tariff rules. These rules often feature vague boundaries and implicit logic relationships, making precise application challenging for agents. However, this critical capability is largely overlooked by current agent benchmarks. To fill this gap, we introduce HSCodeComp, the first realistic, expert-level e-commerce benchmark designed to evaluate deep search agents in hierarchical rule application. In this task, the deep reasoning process of agents is guided by these rules to predict 10-digit Harmonized System Code (HSCode) of products with noisy but realistic descriptions. These codes, established by the World Customs Organization, are vital for global supply chain efficiency. Built from real-world data collected from large-scale e-commerce platforms, our proposed HSCodeComp comprises 632 product entries spanning diverse product categories, with these HSCodes annotated by several human experts. Extensive experimental results on several state-of-the-art LLMs, open-source, and closed-source agents reveal a huge performance gap: best agent achieves only 46.8% 10-digit accuracy, far below human experts at 95.0%. Besides, detailed analysis demonstrates the challenges of hierarchical rule application, and test-time scaling fails to improve performance further.

Figures (17)

• Figure 1: Left: Recent benchmarks reveal the increasing knowledge complexity and capability requirements for agents. Right: 10-digit HSCode accuracy of state-of-the-art baseline largely lags behind human experts (46.8% $<$ 95.0%), proving the challenges of hierarchical rule application. The closed-source agent ($\star$) is evaluated on the subset due to API unavailability.
• Figure 2: One example of a game console product in HSCodeComp.
• Figure 3: The pipeline for human experts to annotate the HSCodes, including two human experts for HSCode annotation (Step 1 to 5) and one additional expert for quality validation (Step 6).
• Figure 4: Failures analysis.
• Figure 5: Both figures show the category distribution on the left blue bars. Left: Challenging Product Distribution (CID). Right: Average Performance Distribution (APD).