Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Domain Adaptation of Foundation LLMs for e-Commerce

Christian Herold, Michael Kozielski, Tala Bazazo, Pavel Petrushkov, Patrycja Cieplicka, Dominika Basaj, Yannick Versley, Seyyed Hadi Hashemi, Shahram Khadivi

TL;DR

This work addresses adapting foundation LLMs to the e-commerce domain through large-scale continued pretraining of Llama-3.1 models (8B and 70B) on a 1 trillion-token, mixed-domain corpus with a strong emphasis on e-commerce data. It systematically analyzes hyperparameters (learning rate, data weighting, context size) and demonstrates that careful CPT can preserve general-domain abilities while significantly boosting e-commerce performance, including multilingual benchmarks where gains reach roughly 25%–30%. A model-merging approach is introduced to precisely balance general and domain-specific capabilities without heavy computation. The results offer a scalable blueprint for industry-focused adaptation of open-domain LLMs to domain-specific needs, with practical implications for e-commerce applications such as product listings and pricing tasks.

Abstract

We present the e-Llama models: 8 billion and 70 billion parameter large language models that are adapted towards the e-commerce domain. These models are meant as foundation models with deep knowledge about e-commerce, that form a base for instruction- and fine-tuning. The e-Llama models are obtained by continuously pretraining the Llama 3.1 base models on 1 trillion tokens of domain-specific data. We discuss our approach and motivate our choice of hyperparameters with a series of ablation studies. To quantify how well the models have been adapted to the e-commerce domain, we define and implement a set of multilingual, e-commerce specific evaluation tasks. We show that, when carefully choosing the training setup, the Llama 3.1 models can be adapted towards the new domain without sacrificing significant performance on general domain tasks. We also explore the possibility of merging the adapted model and the base model for a better control of the performance trade-off between domains.

Domain Adaptation of Foundation LLMs for e-Commerce

TL;DR

This work addresses adapting foundation LLMs to the e-commerce domain through large-scale continued pretraining of Llama-3.1 models (8B and 70B) on a 1 trillion-token, mixed-domain corpus with a strong emphasis on e-commerce data. It systematically analyzes hyperparameters (learning rate, data weighting, context size) and demonstrates that careful CPT can preserve general-domain abilities while significantly boosting e-commerce performance, including multilingual benchmarks where gains reach roughly 25%–30%. A model-merging approach is introduced to precisely balance general and domain-specific capabilities without heavy computation. The results offer a scalable blueprint for industry-focused adaptation of open-domain LLMs to domain-specific needs, with practical implications for e-commerce applications such as product listings and pricing tasks.

Abstract

We present the e-Llama models: 8 billion and 70 billion parameter large language models that are adapted towards the e-commerce domain. These models are meant as foundation models with deep knowledge about e-commerce, that form a base for instruction- and fine-tuning. The e-Llama models are obtained by continuously pretraining the Llama 3.1 base models on 1 trillion tokens of domain-specific data. We discuss our approach and motivate our choice of hyperparameters with a series of ablation studies. To quantify how well the models have been adapted to the e-commerce domain, we define and implement a set of multilingual, e-commerce specific evaluation tasks. We show that, when carefully choosing the training setup, the Llama 3.1 models can be adapted towards the new domain without sacrificing significant performance on general domain tasks. We also explore the possibility of merging the adapted model and the base model for a better control of the performance trade-off between domains.
Paper Structure (20 sections, 4 figures, 5 tables)

This paper contains 20 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Setup, Data and Evaluation
  4. Training Framework and Hardware
  5. Data
  6. Evaluation
  7. Finding the best Setup
  8. Learning Rate
  9. Data Weighting
  10. Context Size
  11. e-Llama
  12. Training
  13. Final models
  14. Model Merging
  15. Conclusion
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Perplexity as a function of the input sequence length for the 8B (e)-Llama-3.0/3.1 models. The 3.0 variants can not handle context sizes much longer than 8k, since they have never seen these lengths in training.
  • Figure 2: Evolution of the model performance over the course of the training. Left: perplexity on heldout data sets in the general/e-commerce domain. Center/Right: Evolution of downstream model performance on English tasks for 8B/70B model.
  • Figure 3: Model merging: 8B Model performance on English general and e-commerce benchmarks as a function of the weight of the e-Llama model parameters vs the base Llama-3.1 model parameters.
  • Figure 4: Llama-3 8B model performance on the 5 e-commerce evaluation tasks as a function of the number of few-shot examples provided in the prompt.