SHERLOCK: Towards Dynamic Knowledge Adaptation in LLM-enhanced E-commerce Risk Management

TL;DR

SHERLOCK addresses the escalating workload of e-commerce risk investigations by integrating a dynamic domain knowledge base, a data flywheel for continual learning, and a Reflect & Refine module to reduce LLM hallucinations. It combines extensible fine-tuning with retrieval-augmented generation to internalize domain knowledge and promptly incorporate new business logic. Key contributions include: 1) a multi-modal domain KB with 1850 entries and 1213 high-quality concepts derived from expert input, 2) a data flywheel that links data labeling, model evaluation, and operations for sustainable adaptation, and 3) the R&R module enabling rapid post-hoc corrections and hotfixes. Experiments on JD.com data show substantial gains in risk localization precision and factual alignment, with online deployment achieving 387% faster investigations and 82% expert acceptance, demonstrating practical impact for dynamic risk environments.

Abstract

The growth of the e-commerce industry has intensified the adversarial dynamics between shadow economy actors and risk management teams. Companies often conduct risk investigations into suspicious cases to identify emerging fraud patterns, thereby enhancing both preemptive risk prevention and post-hoc governance. However, the sheer volume of case analyses imposes a substantial workload on risk management analysts, as each case requires the integration of long-term expert experience and meticulous scrutiny across multiple risk dimensions. Additionally, individual disparities among analysts hinder the establishment of uniform and high-standard workflows. To address these challenges, we propose the SHERLOCK framework, which leverages the reasoning capabilities of large language models (LLMs) to assist analysts in risk investigations. Our approach consists of three primary components: (1) extracting risk management knowledge from multi-modal data and constructing a domain knowledge base (KB), (2) building an intelligent platform guided by the data flywheel paradigm that integrates daily operations, expert annotations, and model evaluations, with iteratively fine-tuning for preference alignment, and (3) introducing a Reflect & Refine (R&R) module that collaborates with the domain KB to establish a rapid response mechanism for evolving risk patterns. Experiments conducted on the real-world transaction dataset from JD dot com demonstrate that our method significantly improves the precision of both factual alignment and risk localization within the LLM analysis results. Deployment of the SHERLOCK-based LLM system on JD dot com has substantially enhanced the efficiency of case investigation workflows for risk managers.

Figures (4)

• Figure 1: Overview of SHERLOCK.
• Figure 2: Illustration of domain Knowledge Base.
• Figure 3: Illustration of Data Flywheel.
• Figure 4: Illustration of R&R Module.