Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering

Vlad Negoita, Mihai Masala, Traian Rebedea

TL;DR

The paper addresses the scarcity and variable quality of Romanian pretraining data for LLMs. It introduces a multidimensional filtering pipeline that uses a lightweight multitask annotator to predict educational value, topic, format, and education level, leveraging a small human-labeled set and a large LLM-labeled corpus. The authors construct FineWeb2-Edu-Ro and demonstrate that filtering improves continual pretraining performance on Romanian benchmarks (RoMMLU, RoARC, RoHellaSwag) and reveals cross-lingual differences in topic distributions. They also discuss limitations related to data size under scaling laws and limited format diversity, proposing future work to broaden data diversity and coverage across formats and domains.

Abstract

Large Language Models (LLMs) have recently exploded in popularity, often matching or outperforming human abilities on many tasks. One of the key factors in training LLMs is the availability and curation of high-quality data. Data quality is especially crucial for under-represented languages, where high-quality corpora are scarce. In this work we study the characteristics and coverage of Romanian pretraining corpora and we examine how they differ from English data. By training a lightweight multitask model on carefully LLM-annotated Romanian texts, we are able to analyze and perform multi-level filtering (e.g., educational value, topic, format) to generate high-quality pretraining datasets. Our experiments show noteworthy trends in the topics present in Romanian and English data, while also proving the effectiveness of filtering data through improved LLM pretraining performance across multiple benchmarks.

Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering

TL;DR

The paper addresses the scarcity and variable quality of Romanian pretraining data for LLMs. It introduces a multidimensional filtering pipeline that uses a lightweight multitask annotator to predict educational value, topic, format, and education level, leveraging a small human-labeled set and a large LLM-labeled corpus. The authors construct FineWeb2-Edu-Ro and demonstrate that filtering improves continual pretraining performance on Romanian benchmarks (RoMMLU, RoARC, RoHellaSwag) and reveals cross-lingual differences in topic distributions. They also discuss limitations related to data size under scaling laws and limited format diversity, proposing future work to broaden data diversity and coverage across formats and domains.

Abstract

Large Language Models (LLMs) have recently exploded in popularity, often matching or outperforming human abilities on many tasks. One of the key factors in training LLMs is the availability and curation of high-quality data. Data quality is especially crucial for under-represented languages, where high-quality corpora are scarce. In this work we study the characteristics and coverage of Romanian pretraining corpora and we examine how they differ from English data. By training a lightweight multitask model on carefully LLM-annotated Romanian texts, we are able to analyze and perform multi-level filtering (e.g., educational value, topic, format) to generate high-quality pretraining datasets. Our experiments show noteworthy trends in the topics present in Romanian and English data, while also proving the effectiveness of filtering data through improved LLM pretraining performance across multiple benchmarks.

Paper Structure

This paper contains 16 sections, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Taxonomy definition
  4. Approach
  5. Training dataset
  6. Model distillation
  7. Results and Discussion
  8. Conclusions
  9. Taxonomy definition
  10. Choosing the LLM
  11. Impact of Training Data over Model Performance
  12. Lightweight Classifier Setup
  13. Impact of Filtering Thresholds
  14. Educational Level Distribution
  15. Training Details
  16. ...and 1 more sections

Figures (8)

  • Figure 1: Flowchart detailing the multi-stage pipeline for building an educational Romanian pretraining dataset. The process includes initial human annotation and LLM benchmarking, distillation of a lightweight annotator for full-scale labeling, and final filtering with a subsequent cross-lingual distribution analysis.
  • Figure 2: Pretraining results using filtered (RoEdu&JQL) and unfiltered FineWeb2 data. Note the consistent improvement in performance when using filtered data, with the best overall results obtained using our proposed filtering approach.
  • Figure 3: Unfiltered texts topic distribution.
  • Figure 4: Filtered texts topic distribution.
  • Figure 5: Educational value scores against training size.
  • ...and 3 more figures