IT5: Text-to-text Pretraining for Italian Language Understanding and Generation

TL;DR

IT5 tackles the need for language-specific encoder-decoder models for Italian by pretraining four IT5 variants on a thoroughly cleaned Italian mC4 corpus and evaluating them with ItaGen, a comprehensive Italian NLP benchmark. The approach demonstrates that monolingual IT5 models achieve state-of-the-art results on several Italian generation and understanding tasks, outperforming multilingual baselines in many cases. Key contributions include the large-scale Italian data cleaning pipeline, the release of four IT5 model sizes, and the ItaGen benchmark that consolidates tasks across summarization, QA, generation, and style transfer. The findings highlight the practical impact of language-specific pretraining for Italian while acknowledging biases from web data and the limits imposed by computational resources. These insights pave the way for further exploration of monolingual versus multilingual pretraining trade-offs in less-resourced languages.

Abstract

We introduce IT5, the first family of encoder-decoder transformer models pretrained specifically on Italian. We document and perform a thorough cleaning procedure for a large Italian corpus and use it to pretrain four IT5 model sizes. We then introduce the ItaGen benchmark, which includes a broad range of natural language understanding and generation tasks for Italian, and use it to evaluate the performance of IT5 models and multilingual baselines. We find monolingual IT5 models to provide the best scale-to-performance ratio across tested models, consistently outperforming their multilingual counterparts and setting a new state-of-the-art for Italian language generation.

Figures (1)

• Figure 1: Loss curves for the masked span prediction task used to pre-train the IT5 models.