Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining

TL;DR

This work expands metadata conditioning for LLM pretraining beyond URL signals, showing that fine-grained metadata (e.g., quality scores, domain information) prepended to text can speed up learning, while coarse signals often do not. It introduces metadata appending as an auxiliary task and learnable meta-tokens, demonstrating both approaches yield notable improvements and offering mechanistic insight via probing into latent representations. Key findings include the primacy of granularity, the lack of additive effects when combining metadata signals, and the ability of LMs to learn quality-aware latent structure through dedicated tokens. The results provide practical guidelines for integrating metadata to boost pretraining efficiency and inform future work on the interpretability and cross-domain implications of metadata in language models.

Abstract

Incorporating metadata in Large Language Models (LLMs) pretraining has recently emerged as a promising approach to accelerate training. However prior work highlighted only one useful signal-URLs, leaving open the question of whether other forms of metadata could yield greater benefits. In this study, we investigate a wider range of metadata types and find other types of metadata, such as fine-grained indicators of document quality that can also accelerate pretraining when prepended. We identify a common feature among effective metadata: they encode information at a finer granularity. We further introduce metadata appending as a means of improving training efficiency, where predicting an appropriate metadata as auxiliary task can help speed up pretraining. In addition, learnable meta-tokens trained with masked loss can recover part of the speedup by inducing quality-aware latent structure. Using probing, we analyze latent representations to understand how metadata shapes learning. Together, these results yield practical guidelines for integrating metadata to improve both the efficiency and effectiveness of LLM pretraining.

Figures (15)

• Figure 1: Our tokenization. Each document begins with a default beginning-of-sequence (<s>) token. For each sequence, metadata is wrapped between beginning-of-context (<boc>) and end-of-context (<eoc>). Depending on the metadata position, the metadata is prepended (illustrated on the left) or appended (illustrated on the right) to the document. If a long document split into multiple sequences, metadata is attached to each one. A 10% dropout of metadata is always performed.
• Figure 2: Pretraining acceleration measured by downstream evaluation performances. DI stands for domain information and QS stands for quality score.
• Figure 3: Comparison of downstream performance when prepending URL or QS-Fine individually versus in combination. Both metadata types are effective on their own, but combining them yields no additive effect on metadata acceleration.
• Figure 4: Probing results on document topic prediction for QS and DI models prepended with fine- and coarse-grained metadata. Models trained with finer-grained metadata generally achieve better performance.
• Figure 5: Averaged attention weights across all attention heads in different layers. A finer-grained per-layer per-attention head attention pattern is provided in Figure \ref{['fig:attention-pattern-url-parts-finegrained']}.