TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese

TL;DR

The paper tackles linguistic inequity in LLMs by presenting two open-source Brazilian Portuguese language models, TTL-160m and TTL-460m, trained from scratch under a tight budget. It details a budget-conscious pre-training pipeline with two BP corpora (Pt-Corpus and Pt-Corpus-Instruct), a customized SentencePiece tokenizer, and decoder-only Transformer architectures derived from Llama-2. The authors show competitive performance on translated benchmarks, release an instruction-tuned TTL-460m-Chat, and quantify energy usage and emissions, highlighting practical trade-offs for low-resource settings. They also discuss limitations (e.g., under-training, sparse BP benchmarks) and outline avenues for future expansion and broader community involvement to advance BP LLM development.

Abstract

Large language models (LLMs) have significantly advanced natural language processing, but their progress has yet to be equal across languages. While most LLMs are trained in high-resource languages like English, multilingual models generally underperform monolingual ones. Additionally, aspects of their multilingual foundation sometimes restrict the byproducts they produce, like computational demands and licensing regimes. In this study, we document the development of open-foundation models tailored for use in low-resource settings, their limitations, and their benefits. This is the TeenyTinyLlama pair: two compact models for Brazilian Portuguese text generation. We release them under the permissive Apache 2.0 license on GitHub and Hugging Face for community use and further development. See https://github.com/Nkluge-correa/TeenyTinyLlama

Figures (4)

• Figure 1: Learning Curves for the TTL pair
• Figure 2: TTL-460 accuracy on the ARC-Challenge during training
• Figure 3: Sample generated by TTL-460m-Chat
• Figure :