Knowledge Graph Completion Models are Few-shot Learners: An Empirical Study of Relation Labeling in E-commerce with LLMs

TL;DR

This study investigates using large language models (PaLM and GPT-3.5) for few-shot relation labeling between product types in e-commerce knowledge graphs. It demonstrates that LLMs outperform traditional KG completion baselines under limited labeled data and can even match or exceed human labeling, aided by informative explanations. The work analyzes prompt engineering and compares LLM labeling to independent and dependent human labeling, showing that explanations can improve human agreement. Overall, the results suggest LLMs offer scalable, language-understanding-driven benefits for e-commerce KG completion and related tasks, potentially reducing reliance on costly human labeling.

Abstract

Knowledge Graphs (KGs) play a crucial role in enhancing e-commerce system performance by providing structured information about entities and their relationships, such as complementary or substitutable relations between products or product types, which can be utilized in recommender systems. However, relation labeling in KGs remains a challenging task due to the dynamic nature of e-commerce domains and the associated cost of human labor. Recently, breakthroughs in Large Language Models (LLMs) have shown surprising results in numerous natural language processing tasks. In this paper, we conduct an empirical study of LLMs for relation labeling in e-commerce KGs, investigating their powerful learning capabilities in natural language and effectiveness in predicting relations between product types with limited labeled data. We evaluate various LLMs, including PaLM and GPT-3.5, on benchmark datasets, demonstrating their ability to achieve competitive performance compared to humans on relation labeling tasks using just 1 to 5 labeled examples per relation. Additionally, we experiment with different prompt engineering techniques to examine their impact on model performance. Our results show that LLMs significantly outperform existing KG completion models in relation labeling for e-commerce KGs and exhibit performance strong enough to replace human labeling.

Figures (2)

• Figure 1: Prompt examples with different principles.
• Figure 2: Comparision of accuracy between KG models and the LLM in the Electornics and Instacart dataset.