LiLiuM: eBay's Large Language Models for e-commerce

TL;DR

LiLiuM presents a fully in-house family of e-commerce focused LLMs (1B, 7B, 13B) developed to provide license, data, vocabulary, and architecture control for eBay. The approach combines a decoder-only transformer with a multilingual, e-commerce–biased data mix and a bespoke tokenizer, achieving competitive English NLU while surpassing LLaMA-2 on non-English NLU, MT, and e-commerce tasks, and delivering faster eBay-specific inference. Key contributions include a detailed data curation pipeline (e-commerce listings, catalog, reviews), a multilingual mixed-data pretraining strategy (RefinedWeb + RedPajama-V2), a domain-specific tokenizer with a 65k vocabulary, structured data serialization methods, and evidence that checkpoint averaging improves final performance. Practically, LiLiuM enables tighter in-house control and better domain adaptation, with concrete speedups and a blueprint for scaling to larger models and more sophisticated data pipelines in the future.

Abstract

We introduce the LiLiuM series of large language models (LLMs): 1B, 7B, and 13B parameter models developed 100% in-house to fit eBay's specific needs in the e-commerce domain. This gives eBay full control over all aspects of the models including license, data, vocabulary, and architecture. We expect these models to be used as a foundation for fine-tuning and instruction-tuning, eliminating dependencies to external models. The LiLiuM LLMs have been trained on 3 trillion tokens of multilingual text from general and e-commerce domain. They perform similar to the popular LLaMA-2 models on English natural language understanding (NLU) benchmarks. At the same time, we outperform LLaMA-2 on non-English NLU tasks, machine translation and on e-commerce specific downstream tasks. As part of our data mixture, we utilize the newly released RedPajama-V2 dataset for training and share our insights regarding data filtering and deduplication. We also discuss in detail how to serialize structured data for use in autoregressive language modeling. We provide insights on the effects of including code and parallel machine translation data in pre-training. Furthermore, we develop our own tokenizer and model vocabulary, customized towards e-commerce. This way, we can achieve up to 34% speed-up in text generation on eBay-specific downstream tasks compared to LLaMA-2. Finally, in relation to LLM pretraining, we show that checkpoint averaging can further improve over the best individual model checkpoint.

Figures (5)

• Figure 1: Overview of the item selection task.
• Figure 2: Overview of the aspect prediction task.
• Figure 3: Percentage of e-commerce data in training vs model performance. 'general domain tasks' indicates the average of zs-1, zs-2, and zs-3 (see Section \ref{['subsec:evaluation']}) and 'e-commerce tasks' indicates the average of aspect prediction and item selection.
• Figure 4: Performance on the downstream English NLU task aggregate (see Section \ref{['subsubsec:evaluation_NLU']}) during training.
• Figure 5: Performance of LiLiuM 7B model checkpoints on the downstream English NLU task aggregate (see Section \ref{['subsubsec:evaluation_NLU']}). Checkpoint averaging indicates averaging the model parameters of all checkpoints between the current and the last checkpoint.