CamemBERT: a Tasty French Language Model

TL;DR

The paper demonstrates the feasibility of training a monolingual RoBERTa‑style language model for French using web‑scale Common Crawl data (OSCAR). CamemBERT achieves state‑of‑the‑art results on POS tagging, dependency parsing, NER, and NLI, using both fine‑tuning and frozen embeddings, and requires surprisingly little data (as little as 4GB) to reach strong performance. It shows that heterogeneous web data can outperform Wikipedia‑only corpora and that model size and pretraining data origin significantly influence outcomes, with smaller, diverse data sometimes matching or exceeding larger datasets. The work also provides openly available models and training data, promoting reproducibility and accessibility for other languages.

Abstract

Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models --in all languages except English-- very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.

Figures (1)

• Figure 1: Impact of number of pretraining steps on downstream performance for CamemBERT.