ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the Portuguese Language

TL;DR

This work introduces $\texttt{ptt5-v2}$, investigating the continued pretraining of T5 models for Portuguese, and explores the effects of different pretraining configurations, including pretraining data quality, optimization strategies, and multi-epoch pretraining.

Abstract

Despite advancements in Natural Language Processing (NLP) and the growing availability of pretrained models, the English language remains the primary focus of model development. Continued pretraining on language-specific corpora provides a practical solution for adapting models to other languages. However, the impact of different pretraining settings on downstream tasks remains underexplored. This work introduces $\texttt{ptt5-v2}$, investigating the continued pretraining of T5 models for Portuguese. We first develop a baseline set of settings and pretrain models with sizes up to 3B parameters. Finetuning on three Portuguese downstream tasks (assin2 STS, assin2 RTE, and TweetSentBR) yields SOTA results on the latter two. We then explore the effects of different pretraining configurations, including pretraining data quality, optimization strategies, and multi-epoch pretraining. Perhaps surprisingly, their impact remains subtle compared to our baseline. We release $\texttt{ptt5-v2}$ pretrained checkpoints and their MonoT5-based finetuned $\texttt{MonoPTT5}$ rerankers on HuggingFace in their respective collections at \url{https://huggingface.co/unicamp-dl}.

Figures (8)

• Figure 1: NPM vs parameters for varying T5 configurations.
• Figure 2: Effect of pretraining data quality on downstream task performance.
• Figure 3: Learning rate schedules: Constant vs. inverse square root scheduler as a function of training steps.
• Figure 4: Effect of scheduler used during pretraining. Epochs are relative to mC4-pt.
• Figure 5: Retrieval results on mMARCO-pt. mColbert and mT5 values are from Bonifacio et al. bonifacio2022mmarco. Total size excludes embedding parameters.