EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data

TL;DR

This work tackles adapting LLMs to the e-commerce domain through continual pre-training on a mix of general and e-commerce unlabeled data, supplemented by a data-mixing strategy for semi-structured data. It introduces EcomGPT-CT based on BLOOM, along with benchmarks for few-shot in-context learning and zero-shot performance after instruction tuning in e-commerce. Empirical results show domain-specific continual pre-training improves e-commerce tasks, and the proposed data-mixing approach yields additional gains, while general NLP performance is largely preserved when a balanced data mix is used. The study provides practical guidance for resource-constrained domain adaptation of LLMs and motivates further research into sophisticated data utilization and domain-specific benchmarks.

Abstract

Large Language Models (LLMs) pre-trained on massive corpora have exhibited remarkable performance on various NLP tasks. However, applying these models to specific domains still poses significant challenges, such as lack of domain knowledge, limited capacity to leverage domain knowledge and inadequate adaptation to domain-specific data formats. Considering the exorbitant cost of training LLMs from scratch and the scarcity of annotated data within particular domains, in this work, we focus on domain-specific continual pre-training of LLMs using E-commerce domain as an exemplar. Specifically, we explore the impact of continual pre-training on LLMs employing unlabeled general and E-commercial corpora. Furthermore, we design a mixing strategy among different data sources to better leverage E-commercial semi-structured data. We construct multiple tasks to assess LLMs' few-shot In-context Learning ability and their zero-shot performance after instruction tuning in E-commerce domain. Experimental results demonstrate the effectiveness of continual pre-training of E-commerce LLMs and the efficacy of our devised data mixing strategy.

Figures (3)

• Figure 1: General framework of model training.
• Figure 2: An example of our dataset mixing strategy.
• Figure 3: Performance variation of EcomGPT-CT-7B during the continual pre-training process.