Training LLMs on HPC Systems: Best Practices from the OpenGPT-X Project

TL;DR

The paper addresses the challenge of training open, multilingual LLMs on European HPC infrastructure by detailing the end-to-end workflow used to train Teuken-7B on JUWELS Booster. It presents a practical synthesis of hardware choices, software stack evolution (Megatron-LM with modern optimizations), data storage strategies, and a benchmarking and profiling regime to optimize large-scale training. Key contributions include the setup repository for cross-cluster deployment, integration with profiling and benchmarking tools (CARAML, JUPITER), and lessons learned for reliability, reproducibility, and operational efficiency. The work demonstrates the feasibility and value of an open, sovereign AI training pipeline in Europe, informing future exascale deployments and infrastructure-aware ML workflows.

Abstract

The training of large language models (LLMs) requires substantial computational resources, complex software stacks, and carefully designed workflows to achieve scalability and efficiency. This report presents best practices and insights gained from the OpenGPT-X project, a German initiative focused on developing open, multilingual LLMs optimized for European languages. We detail the use of high-performance computing (HPC) systems, primarily JUWELS Booster at JSC, for training Teuken-7B, a 7-billion-parameter transformer model. The report covers system architecture, training infrastructure, software choices, profiling and benchmarking tools, as well as engineering and operational challenges.