Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service

Haoyu He, Jinyu Zhuang, Haoran Chu, Shuhang Yu, J, T AI Group, Hao Wang, Kunpeng Han

Abstract

Multilingual intent classification is central to customer-service systems on global logistics platforms, where models must process noisy user queries across languages and hierarchical label spaces. Yet most existing multilingual benchmarks rely on machine-translated text, which is typically cleaner and more standardized than native customer requests and can therefore overestimate real-world robustness. We present a public benchmark for hierarchical multilingual intent classification constructed from real logistics customer-service logs. The dataset contains approximately 30K de-identified, stand-alone user queries curated from 600K historical records through filtering, LLM-assisted quality control, and human verification, and is organized into a two-level taxonomy with 13 parent and 17 leaf intents. English, Spanish, and Arabic are included as seen languages, while Indonesian, Chinese, and additional test-only languages support zero-shot evaluation. To directly measure the gap between synthetic and real evaluation, we provide paired native and machine-translated test sets and benchmark multilingual encoders, embedding models, and small language models under flat and hierarchical protocols. Results show that translated test sets substantially overestimate performance on noisy native queries, especially for long-tail intents and cross-lingual transfer, underscoring the need for more realistic multilingual intent benchmarks.

From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service

Abstract

Multilingual intent classification is central to customer-service systems on global logistics platforms, where models must process noisy user queries across languages and hierarchical label spaces. Yet most existing multilingual benchmarks rely on machine-translated text, which is typically cleaner and more standardized than native customer requests and can therefore overestimate real-world robustness. We present a public benchmark for hierarchical multilingual intent classification constructed from real logistics customer-service logs. The dataset contains approximately 30K de-identified, stand-alone user queries curated from 600K historical records through filtering, LLM-assisted quality control, and human verification, and is organized into a two-level taxonomy with 13 parent and 17 leaf intents. English, Spanish, and Arabic are included as seen languages, while Indonesian, Chinese, and additional test-only languages support zero-shot evaluation. To directly measure the gap between synthetic and real evaluation, we provide paired native and machine-translated test sets and benchmark multilingual encoders, embedding models, and small language models under flat and hierarchical protocols. Results show that translated test sets substantially overestimate performance on noisy native queries, especially for long-tail intents and cross-lingual transfer, underscoring the need for more realistic multilingual intent benchmarks.
Paper Structure (27 sections, 4 figures, 7 tables)

This paper contains 27 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Dataset Construction
  3. Construction Pipeline
  4. Source data.
  5. De-identification and cleaning.
  6. Taxonomy construction.
  7. Annotation and quality control.
  8. Dataset Characteristics
  9. Overview
  10. Native vs. Synthetic Evaluation
  11. Hierarchical Labels, Long-Tail Sparsity, and Noise
  12. Experiments
  13. Benchmark Tasks and Evaluation Protocol
  14. Baselines
  15. Results
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Construction pipeline for the benchmark.
  • Figure 2: Top-$k$ label coverage for the parent and leaf tasks. The 17-way leaf setting is substantially more long-tailed than the 13-way parent setting.
  • Figure 3: Comprehensive evaluation of clustering configurations. (Left) Normalized overlay of six clustering metrics across $K \in \{10, \dots, 19\}$. All metrics are scaled to $[0, 1]$ and direction-aligned so that higher values indicate better clustering quality. The black line represents the aggregate score, which peaks at $K=13$ (indicated by the red dashed line). (Right) Radar chart comparing the multivariate performance profiles of the top-six $K$ candidates. $K=13$ achieves the highest aggregate score ($0.672$) and exhibits the most balanced, well-rounded performance across the evaluated dimensions.
  • Figure 4: Representative XML-style system prompt used for generation-based intent classification.