LLMBox: A Comprehensive Library for Large Language Models

TL;DR

LLMBox tackles reproducibility and fair comparison in large language model research by delivering a unified library for training, utilization, and evaluation. It introduces a unified data interface, a broad evaluation suite across 18 tasks and 56 datasets, and practical usability features (GPU memory calculator, prefix caching, packing) to streamline experimentation. The framework supports diverse training methods (pre-training, instruction tuning, human alignment) with efficiency mechanisms (LoRA/QLoRA, DeepSpeed, packing) and domain adaptation capabilities, validated by extensive experiments demonstrating effectiveness and efficiency. By enabling reproducible pipelines and cross-model comparisons, LLMBox stands to accelerate research and deployment of LLMs.

Abstract

To facilitate the research on large language models (LLMs), this paper presents a comprehensive and unified library, LLMBox, to ease the development, use, and evaluation of LLMs. This library is featured with three main merits: (1) a unified data interface that supports the flexible implementation of various training strategies, (2) a comprehensive evaluation that covers extensive tasks, datasets, and models, and (3) more practical consideration, especially on user-friendliness and efficiency. With our library, users can easily reproduce existing methods, train new models, and conduct comprehensive performance comparisons. To rigorously test LLMBox, we conduct extensive experiments in a diverse coverage of evaluation settings, and experimental results demonstrate the effectiveness and efficiency of our library in supporting various implementations related to LLMs. The detailed introduction and usage guidance can be found at https://github.com/RUCAIBox/LLMBox.

Figures (2)

• Figure 1: The overall framework of our LLMBox, supporting the training, utilization and evaluation of LLMs.
• Figure 2: Usage examples of our LLMBox library on six representative tasks.