CodeTF: One-stop Transformer Library for State-of-the-art Code LLMs

TL;DR

CodeTF tackles the fragmented Code LLM ecosystem by delivering a unified, open-source library that streamlines data preparation, training, serving, and evaluation for diverse Code LLMs. It combines a unified loading/serving interface, parameter-efficient fine-tuning, a comprehensive code-utility suite with AST parsing for 15+ languages, and standardized benchmarking with metrics across multiple datasets. Its six-module design (Model Zoo, Serving, Training, Data Utility, Code Utility, Evaluator) enables end-to-end workflows with reproducible results and efficient deployment through 8-bit/$4$-bit quantization and PEFT methods like LoRA and Prefix-Tuning. By reducing engineering overhead and providing a transparent, extensible platform, CodeTF accelerates both research and practical deployment of state-of-the-art code intelligence solutions.

Abstract

Code intelligence plays a key role in transforming modern software engineering. Recently, deep learning-based models, especially Transformer-based large language models (LLMs), have demonstrated remarkable potential in tackling these tasks by leveraging massive open-source code data and programming language features. However, the development and deployment of such models often require expertise in both machine learning and software engineering, creating a barrier for the model adoption. In this paper, we present CodeTF, an open-source Transformer-based library for state-of-the-art Code LLMs and code intelligence. Following the principles of modular design and extensible framework, we design CodeTF with a unified interface to enable rapid access and development across different types of models, datasets and tasks. Our library supports a collection of pretrained Code LLM models and popular code benchmarks, including a standardized interface to train and serve code LLMs efficiently, and data features such as language-specific parsers and utility functions for extracting code attributes. In this paper, we describe the design principles, the architecture, key modules and components, and compare with other related library tools. Finally, we hope CodeTF is able to bridge the gap between machine learning/generative AI and software engineering, providing a comprehensive open-source solution for developers, researchers, and practitioners.

Figures (5)

• Figure 1: An overview of CodeTF: A comprehensive Transformer-based library supporting the complete workflow for Code LLMs---from data preparation and model training to serving and evaluation. The library integrates pretrained models, code utilities for 15+ programming languages, and standardized benchmarks into a unified framework.
• Figure 2: Common workflows when working with Code LLMs. Practitioners typically need to handle four interconnected tasks: data preparation, model training/fine-tuning, model serving, and evaluation. CodeTF provides unified support for all four stages.
• Figure 3: System architecture of CodeTF showing the six core modules: Model Zoo (codetf.model), Model Serving (codetf.predict), Model Training (codetf.trainer), Data Utility (codetf.data_utility), Code Utility (codetf.code_utility), and Evaluator (codetf.performance).
• Figure 4: Model loading pipeline in CodeTF. The load_model_pipeline function provides a unified entry point that dispatches to architecture-specific model classes (CausalLMModel, Seq2SeqModel, BERTModel) based on the requested model type.
• Figure 5: Training module architecture. The BaseTrainer class provides common functionality (checkpoint saving, logging, GPU management) inherited by architecture-specific trainers (CausalLMTrainer, Seq2SeqTrainer, BERTTrainer).