LLM4Tag: Automatic Tagging System for Information Retrieval via Large Language Models

TL;DR

This paper introduces LLM4Tag, a three-module tagging framework designed to enhance information retrieval with Large Language Models: graph-based tag recall for complete candidate tag sets, knowledge-enhanced tag generation that injects long-term and short-term domain knowledge, and tag confidence calibration to produce reliable relevance scores. The graph recall constructs a content-tag semantic graph using deterministic and similarity edges and meta-paths to recall tags beyond simple matches. Knowledge injection combines supervised fine-tuning with continual, retrieval-based guidance to adapt to evolving domain knowledge. Confidence calibration produces probabilistic tag relevance and allows pruning to balance accuracy and coverage, enabling robust production deployment. Across three large industrial datasets, LLM4Tag achieves state-of-the-art performance and has been deployed online, serving hundreds of millions of users, highlighting practical impact in scalable information retrieval systems.

Abstract

Tagging systems play an essential role in various information retrieval applications such as search engines and recommender systems. Recently, Large Language Models (LLMs) have been applied in tagging systems due to their extensive world knowledge, semantic understanding, and reasoning capabilities. Despite achieving remarkable performance, existing methods still have limitations, including difficulties in retrieving relevant candidate tags comprehensively, challenges in adapting to emerging domain-specific knowledge, and the lack of reliable tag confidence quantification. To address these three limitations above, we propose an automatic tagging system LLM4Tag. First, a graph-based tag recall module is designed to effectively and comprehensively construct a small-scale highly relevant candidate tag set. Subsequently, a knowledge-enhanced tag generation module is employed to generate accurate tags with long-term and short-term knowledge injection. Finally, a tag confidence calibration module is introduced to generate reliable tag confidence scores. Extensive experiments over three large-scale industrial datasets show that LLM4Tag significantly outperforms the state-of-the-art baselines and LLM4Tag has been deployed online for content tagging to serve hundreds of millions of users.

Figures (9)

• Figure 1: LLM-enhanced tagging systems and their three limitations. (L1) Simple match-based recall is prone to missing relevant tags; (L2) The emerging domain-specific knowledge may not align with the pre-trained knowledge of LLMs; (L3) LLMs cannot accurately quantify tag confidence.
• Figure 2: The overall framework of LLM4Tag architecture of LLM4Tag, consisting of three modules: graph-based tag recall module, knowledge-enhanced tag generation module, and tag confidence calibration module.
• Figure 3: Prompt template for basic tag generation in advertisement creatives tagging scenario.
• Figure 4: Prompt template for retrieval enhanced tag generation in advertisement creatives tagging scenario.
• Figure 5: Prompt template for tag confidence judgment in advertisement creatives tagging scenario.