Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TigerBot: An Open Multilingual Multitask LLM

Ye Chen, Wei Cai, Liangmin Wu, Xiaowei Li, Zhanxuan Xin, Cong Fu

TL;DR

TigerBot presents a publicly released family of open-source multilingual LLMs (7B–180B) built on Llama-2 and BLOOM, featuring a comprehensive training stack, scalable infrastructure, and toolchains aimed at democratizing LLM development. The authors introduce a holistic training regime (SFT, RLHF with DPO), a 3D Megatron-DeepSpeed parallelism framework, and enhancements in tokenizer, long-context handling, and safety, achieving competitive gains over open-source baselines and enabling practical applications. They validate the approach with multi-language benchmarks and extensive evaluations, and demonstrate diverse applications including long-context QA, recursive summarization, function calling, online search, role-playing, and intelligent hardware integration. The work emphasizes open dissemination, efficiency, and real-world applicability, while acknowledging challenges in reliability, infrastructure sustainability, and broad deployment in real-world tasks.

Abstract

We release and introduce the TigerBot family of large language models (LLMs), consisting of base and chat models, sized from 7, 13, 70 and 180 billion parameters. We develop our models embarking from Llama-2 and BLOOM, and push the boundary further in data, training algorithm, infrastructure, and application tools. Our models yield meaningful performance gain over SOTA open-source models, e.g., Llama-2, specifically 6% gain in English and 20% gain in Chinese. TigerBot model family also achieves leading performance in major academic and industrial benchmarks and leaderboards. We believe that TigerBot represents just a snapshot of lightning-fast progression in LLM open-source community. Therefore, we are thrilled to give back by publicly releasing our models and reporting our approach behind, with additional emphases on building SOTA LLMs in a democratized way and making LLMs of use in real-world applications.

TigerBot: An Open Multilingual Multitask LLM

TL;DR

TigerBot presents a publicly released family of open-source multilingual LLMs (7B–180B) built on Llama-2 and BLOOM, featuring a comprehensive training stack, scalable infrastructure, and toolchains aimed at democratizing LLM development. The authors introduce a holistic training regime (SFT, RLHF with DPO), a 3D Megatron-DeepSpeed parallelism framework, and enhancements in tokenizer, long-context handling, and safety, achieving competitive gains over open-source baselines and enabling practical applications. They validate the approach with multi-language benchmarks and extensive evaluations, and demonstrate diverse applications including long-context QA, recursive summarization, function calling, online search, role-playing, and intelligent hardware integration. The work emphasizes open dissemination, efficiency, and real-world applicability, while acknowledging challenges in reliability, infrastructure sustainability, and broad deployment in real-world tasks.

Abstract

We release and introduce the TigerBot family of large language models (LLMs), consisting of base and chat models, sized from 7, 13, 70 and 180 billion parameters. We develop our models embarking from Llama-2 and BLOOM, and push the boundary further in data, training algorithm, infrastructure, and application tools. Our models yield meaningful performance gain over SOTA open-source models, e.g., Llama-2, specifically 6% gain in English and 20% gain in Chinese. TigerBot model family also achieves leading performance in major academic and industrial benchmarks and leaderboards. We believe that TigerBot represents just a snapshot of lightning-fast progression in LLM open-source community. Therefore, we are thrilled to give back by publicly releasing our models and reporting our approach behind, with additional emphases on building SOTA LLMs in a democratized way and making LLMs of use in real-world applications.
Paper Structure (23 sections, 1 equation, 6 figures, 4 tables)

This paper contains 23 sections, 1 equation, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. TigerBot models
  3. Training data
  4. Training method
  5. Tokenizer
  6. Training framework
  7. Holistic training
  8. Supervised fine-tuning (SFT)
  9. Reinforcement learning with human feedback (RLHF)
  10. Long sequence
  11. Quantization
  12. Safety
  13. Hyperparameters
  14. Training hardware
  15. Evaluation
  16. ...and 8 more sections

Figures (6)

  • Figure 1: Training loss for Tigerbot models
  • Figure 2: Training data proportions
  • Figure 3: Training efficiency for Tigerbot models
  • Figure 4: Tigerbot-70b-SFT loss with different batch size
  • Figure 5: A scalable approach to Tigerbot's role-playing LLM
  • ...and 1 more figures