Sabiá: Portuguese Large Language Models

TL;DR

The study challenges the notion that a single multilingual model suffices by showing that monolingual pretraining on Portuguese, even with a small fraction of the original budget, substantially boosts performance on Portuguese tasks. By extending English-centric models (LLaMA and GPT-J) with Portuguese data, the authors build Sabiá and Sabiá-J and evaluate them on the Poeta benchmark, where Sabiá-65B reaches parity with GPT-3.5-turbo. Results indicate that gains are driven largely by domain-specific and language-specific knowledge learned during monolingual pretraining, with native datasets benefiting more than translated ones. The work suggests a shift toward language- and domain-specific specialist models as a cost-effective path to high performance in targeted languages and tasks.

Abstract

As the capabilities of language models continue to advance, it is conceivable that "one-size-fits-all" model will remain as the main paradigm. For instance, given the vast number of languages worldwide, many of which are low-resource, the prevalent practice is to pretrain a single model on multiple languages. In this paper, we add to the growing body of evidence that challenges this practice, demonstrating that monolingual pretraining on the target language significantly improves models already extensively trained on diverse corpora. More specifically, we further pretrain GPT-J and LLaMA models on Portuguese texts using 3% or less of their original pretraining budget. Few-shot evaluations on Poeta, a suite of 14 Portuguese datasets, reveal that our models outperform English-centric and multilingual counterparts by a significant margin. Our best model, Sabiá-65B, performs on par with GPT-3.5-turbo. By evaluating on datasets originally conceived in the target language as well as translated ones, we study the contributions of language-specific pretraining in terms of 1) capturing linguistic nuances and structures inherent to the target language, and 2) enriching the model's knowledge about a domain or culture. Our results indicate that the majority of the benefits stem from the domain-specific knowledge acquired through monolingual pretraining.