FineFreq: A Multilingual Character Frequency Dataset from Web-Scale Text
Binbin Xu
TL;DR
The paper tackles the shortage of large-scale, temporally resolved multilingual character-frequency data. It introduces FineFreq, a dataset derived from web-scale corpora (FineWeb and FineWeb2) that covers over 1900 languages from 2013 to 2025 and contains more than 96 trillion characters. The resource provides per-language aggregate and yearly character frequencies with Unicode metadata in CSV and Parquet formats, enabling detailed diachronic analyses and downstream NLP or typographic tasks. By preserving cross-script usage and avoiding aggressive filtering, FineFreq reflects real-world multilingual writing and is publicly available to support reproducibility and broad research use.
Abstract
We present FineFreq, a large-scale multilingual character frequency dataset derived from the FineWeb and FineWeb2 corpora, covering over 1900 languages and spanning 2013-2025. The dataset contains frequency counts for 96 trillion characters processed from 57 TB of compressed text. For each language, FineFreq provides per-character statistics with aggregate and year-level frequencies, allowing fine-grained temporal analysis. The dataset preserves naturally occurring multilingual features such as cross-script borrowings, emoji, and acronyms without applying artificial filtering. Each character entry includes Unicode metadata (category, script, block), enabling domain-specific or other downstream filtering and analysis. The full dataset is released in both CSV and Parquet formats, with associated metadata, available on GitHub and HuggingFace. https://github.com/Bin-2/FineFreq