A Technical Report on the Second Place Solution for the CIKM 2025 AnalytiCup Competition

TL;DR

The paper tackles multilingual category relevance judgment in industrial e-commerce search by replacing complex ensembles with a single-model framework. It combines Chain-of-Thought prompting to decompose decisions into four interpretable steps (translation, intent understanding, category matching, relevance judgment) with LoRA-based fine-tuning on the Qwen2.5-14B base model, enabling parameter-efficient learning. Empirically, it achieves competitive scores on the CIKM 2025 AnalytiCup Proposals (public $0.8902$ and private $0.8889$) while processing $20$ samples per second on an NVIDIA $A100$—demonstrating strong accuracy and throughput. This approach highlights that explicit reasoning within a single large model, when guided by structured prompts and lightweight adaptation, can match or surpass ensemble systems, offering interpretable, scalable solutions for industrial AI.

Abstract

In this work, we address the challenge of multilingual category relevance judgment in e-commerce search, where traditional ensemble-based systems improve accuracy but at the cost of heavy training, inference, and maintenance complexity. To overcome this limitation, we propose a simplified yet effective framework that leverages prompt engineering with Chain-of-Thought task decomposition to guide reasoning within a single large language model. Specifically, our approach decomposes the relevance judgment process into four interpretable subtasks: translation, intent understanding, category matching, and relevance judgment -- and fine-tunes a base model (Qwen2.5-14B) using Low-Rank Adaptation (LoRA) for efficient adaptation. This design not only reduces computational and storage overhead but also enhances interpretability by explicitly structuring the model's reasoning path. Experimental results show that our single-model framework achieves competitive accuracy and high inference efficiency, processing 20 samples per second on a single A100 GPU. In the CIKM 2025 AnalytiCup Competition Proposals, our method achieved 0.8902 on the public leaderboard and 0.8889 on the private leaderboard, validating the effectiveness and robustness of the proposed approach. These results highlight that structured prompting combined with lightweight fine-tuning can outperform complex ensemble systems, offering a new paradigm for scalable industrial AI applications.

Figures (2)

• Figure 1: Prompt template design for the Query-Item Relevance (QI) task.
• Figure 2: Prompt template design for the Query-Category Relevance (QC) task.