asLLR: LLM based Leads Ranking in Auto Sales

TL;DR

asLLR tackles the problem of ranking auto sales leads by jointly modeling tabular customer features and long textual interactions using a decoder-only LLM. It integrates CTR loss and QA loss to train the model, and employs LoRA for efficient fine-tuning, enabling effective handling of both structured and unstructured data. The authors introduce a large-scale automotive leads dataset (300k training, 40k testing) and demonstrate a strong elevation in AUC to 0.8127, with an online sale conversion lift of about 9.5% in A/B tests. To address extremely long text inputs, they add a text summarization module that compresses inputs by roughly 60% and improves performance for long-context scenarios. Overall, asLLR offers a practical approach for business intelligence in auto sales by leveraging LLM-based lead ranking and natural language feature extraction.

Abstract

In the area of commercial auto sales system, high-quality lead score sequencing determines the priority of a sale's work and is essential for optimizing the efficiency of the sales system. Since CRM (Customer Relationship Management) system contains plenty of textual interaction features between sales and customers, traditional techniques such as Click Through Rate (CTR) prediction struggle with processing the complex information inherent in natural language features, which limits their effectiveness in sales lead ranking. Bridging this gap is critical for enhancing business intelligence and decision-making. Recently, the emergence of large language models (LLMs) has opened new avenues for improving recommendation systems, this study introduces asLLR (LLM-based Leads Ranking in Auto Sales), which integrates CTR loss and Question Answering (QA) loss within a decoder-only large language model architecture. This integration enables the simultaneous modeling of both tabular and natural language features. To verify the efficacy of asLLR, we constructed an innovative dataset derived from the customer lead pool of a prominent new energy vehicle brand, with 300,000 training samples and 40,000 testing samples. Our experimental results demonstrate that asLLR effectively models intricate patterns in commercial datasets, achieving the AUC of 0.8127, surpassing traditional CTR estimation methods by 0.0231. Moreover, asLLR enhances CTR models when used for extracting text features by 0.0058. In real-world sales scenarios, after rigorous online A/B testing, asLLR increased the sales volume by about 9.5% compared to the traditional method, providing a valuable tool for business intelligence and operational decision-making.

Figures (9)

• Figure 1: Overview of asLLR framework.
• Figure 2: Methodology for Dataset Construction.
• Figure 3: The Impact of Model Parameter Size on Performance.
• Figure 4: The Influence of Training Data Volume on Model Performance.
• Figure 5: The Impact of Training Settings on asLLR.