Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratch

Zhengzhong Liu, Bowen Tan, Hongyi Wang, Willie Neiswanger, Tianhua Tao, Haonan Li, Fajri Koto, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Liqun Ma, Liping Tang, Nikhil Ranjan, Yonghao Zhuang, Guowei He, Renxi Wang, Mingkai Deng, Robin Algayres, Yuanzhi Li, Zhiqiang Shen, Preslav Nakov, Eric Xing

TL;DR

<3-5 sentence high-level summary>

Abstract

We detail the training of the LLM360 K2-65B model, scaling up our 360-degree OPEN SOURCE approach to the largest and most powerful models under project LLM360. While open-source LLMs continue to advance, the answer to "How are the largest LLMs trained?" remains unclear within the community. The implementation details for such high-capacity models are often protected due to business considerations associated with their high cost. This lack of transparency prevents LLM researchers from leveraging valuable insights from prior experience, e.g., "What are the best practices for addressing loss spikes?" The LLM360 K2 project addresses this gap by providing full transparency and access to resources accumulated during the training of LLMs at the largest scale. This report highlights key elements of the K2 project, including our first model, K2 DIAMOND, a 65 billion-parameter LLM that surpasses LLaMA-65B and rivals LLaMA2-70B, while requiring fewer FLOPs and tokens. We detail the implementation steps and present a longitudinal analysis of K2 DIAMOND's capabilities throughout its training process. We also outline ongoing projects such as TXT360, setting the stage for future models in the series. By offering previously unavailable resources, the K2 project also resonates with the 360-degree OPEN SOURCE principles of transparency, reproducibility, and accessibility, which we believe are vital in the era of resource-intensive AI research.

LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratch

TL;DR

<3-5 sentence high-level summary>

Abstract

We detail the training of the LLM360 K2-65B model, scaling up our 360-degree OPEN SOURCE approach to the largest and most powerful models under project LLM360. While open-source LLMs continue to advance, the answer to "How are the largest LLMs trained?" remains unclear within the community. The implementation details for such high-capacity models are often protected due to business considerations associated with their high cost. This lack of transparency prevents LLM researchers from leveraging valuable insights from prior experience, e.g., "What are the best practices for addressing loss spikes?" The LLM360 K2 project addresses this gap by providing full transparency and access to resources accumulated during the training of LLMs at the largest scale. This report highlights key elements of the K2 project, including our first model, K2 DIAMOND, a 65 billion-parameter LLM that surpasses LLaMA-65B and rivals LLaMA2-70B, while requiring fewer FLOPs and tokens. We detail the implementation steps and present a longitudinal analysis of K2 DIAMOND's capabilities throughout its training process. We also outline ongoing projects such as TXT360, setting the stage for future models in the series. By offering previously unavailable resources, the K2 project also resonates with the 360-degree OPEN SOURCE principles of transparency, reproducibility, and accessibility, which we believe are vital in the era of resource-intensive AI research.
Paper Structure (116 sections, 7 equations, 20 figures, 22 tables)

This paper contains 116 sections, 7 equations, 20 figures, 22 tables.

Table of Contents

  1. Introduction
  2. 360° Open Source Embraces Reproducible, Transparent, and Accessible.
  3. 360° Open Source of the Entire LLM Lifecycle.
  4. Building Powerful 360° Open Source Model.
  5. Overview
  6. Our Open Source Approach
  7. Licenses.
  8. Streamlined Released Artifacts.
  9. K2 Diamond Galleries.
  10. K2 Diamond Loss Spikes.
  11. Fine-Tuning Artifacts.
  12. Pretraining
  13. Pretraining Data
  14. Data Curation: TxT360
  15. Web Data Curation.
  16. ...and 101 more sections

Figures (20)

  • Figure 1: K2 Project Scales up the LLM360 Principles with Richer Artifacts
  • Figure 2: Data mix in K2 Diamond pretraining: major stage (left) and long-context stage (right). In the major stage, Paper data includes ArXiv from RedPajama together2023redpajama and S2ORC Lo2020S2ORCTS. USPTO gao2020pile and Pile-of-law hendersonkrass2022pileoflaw are used as Patent and Law domain texts, respectively. In stage 2, SimpleWiki from Dolma soldaini2024dolma is added into Wikipedia. Math data includes Algebraic-Stack azerbayev2023llemma and Open-Web-Math paster2023openwebmath. Paper data consists of ArXiv together2023redpajama, S2ORC Lo2020S2ORCTS, and PES2O peS2o.
  • Figure 3: An illustration of data sampling strategy of K2 Diamond pretraining.
  • Figure 4: An example of benign spikes during pretraining.
  • Figure 6: An illustration of the hybrid parallelism strategies tuned for K2 Diamond pretraining. Context parallelism, which is simply applied along with the TP parallelism group, is not illustrated in the figure.
  • ...and 15 more figures