Building a Large Japanese Web Corpus for Large Language Models

TL;DR

The paper tackles the lack of high-quality Japanese text in multilingual corpora by constructing a large-scale, high-quality Japanese web corpus from 21 Common Crawl snapshots. It introduces a three-stage pipeline—text extraction, rigorous quality/deduplication/host filtering, and cleaning—to produce a 312.1B-character corpus spanning ~173M pages. Through continual pre-training on multiple base LLMs, the authors demonstrate consistent 6.6–8.1 point improvements on Japanese benchmarks, achieving state-of-the-art performance across model sizes. They also release the models and emphasize open, reproducible research, highlighting the corpus’s practical impact for building capable Japanese LLMs while noting safety and evaluation considerations for future work.

Abstract

Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the quality of Japanese texts. This study builds a large Japanese web corpus by extracting and refining text from the Common Crawl archive (21 snapshots of approximately 63.4 billion pages crawled between 2020 and 2023). This corpus consists of approximately 312.1 billion characters (approximately 173 million pages), which is the largest of all available training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8 billion characters), mC4 (approximately 239.7 billion characters) and OSCAR 23.10 (approximately 74 billion characters). To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct as base LLMs and gained consistent (6.6-8.1 points) improvements on Japanese benchmark datasets. We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora.

Figures (1)

• Figure 1: The pipeline for building a large Japanese web corpus.