Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Model

Peng Di, Jianguo Li, Hang Yu, Wei Jiang, Wenting Cai, Yang Cao, Chaoyu Chen, Dajun Chen, Hongwei Chen, Liang Chen, Gang Fan, Jie Gong, Zi Gong, Wen Hu, Tingting Guo, Zhichao Lei, Ting Li, Zheng Li, Ming Liang, Cong Liao, Bingchang Liu, Jiachen Liu, Zhiwei Liu, Shaojun Lu, Min Shen, Guangpei Wang, Huan Wang, Zhi Wang, Zhaogui Xu, Jiawei Yang, Qing Ye, Gehao Zhang, Yu Zhang, Zelin Zhao, Xunjin Zheng, Hailian Zhou, Lifu Zhu, Xianying Zhu

TL;DR

CodeFuse-13B presents an open-source, multilingual code LLM trained on a rigorously filtered, large-scale code corpus with English and Chinese prompts across 40+ languages. It introduces a comprehensive data preparation and code-analytic pipeline (including Sparrow-based program analysis and code semantic extraction) and employs RoPE and parallel attention to boost efficiency, complemented by SFT and MFT finetuning. Evaluations on HumanEval-x, CodefuseEval, and real-world industry feedback show competitive performance (HumanEval pass@1 of 37.10% for CodeFuse-13B-SFT) and superior handling of Chinese prompts in code generation, translation, comments, and testcase generation. The work emphasizes practical deployment within AntGroup, introduces deployment optimizations (quantization, TensorRT-LLM, caching), and outlines future plans to broaden community resources through MFTCoder, Sparrow, and extended benchmarks.

Abstract

Code Large Language Models (Code LLMs) have gained significant attention in the industry due to their wide applications in the full lifecycle of software engineering. However, the effectiveness of existing models in understanding non-English inputs for multi-lingual code-related tasks is still far from well studied. This paper introduces CodeFuse-13B, an open-sourced pre-trained code LLM. It is specifically designed for code-related tasks with both English and Chinese prompts and supports over 40 programming languages. CodeFuse achieves its effectiveness by utilizing a high quality pre-training dataset that is carefully filtered by program analyzers and optimized during the training process. Extensive experiments are conducted using real-world usage scenarios, the industry-standard benchmark HumanEval-x, and the specially designed CodeFuseEval for Chinese prompts. To assess the effectiveness of CodeFuse, we actively collected valuable human feedback from the AntGroup's software development process where CodeFuse has been successfully deployed. The results demonstrate that CodeFuse-13B achieves a HumanEval pass@1 score of 37.10%, positioning it as one of the top multi-lingual code LLMs with similar parameter sizes. In practical scenarios, such as code generation, code translation, code comments, and testcase generation, CodeFuse performs better than other models when confronted with Chinese prompts.

CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Model

TL;DR

CodeFuse-13B presents an open-source, multilingual code LLM trained on a rigorously filtered, large-scale code corpus with English and Chinese prompts across 40+ languages. It introduces a comprehensive data preparation and code-analytic pipeline (including Sparrow-based program analysis and code semantic extraction) and employs RoPE and parallel attention to boost efficiency, complemented by SFT and MFT finetuning. Evaluations on HumanEval-x, CodefuseEval, and real-world industry feedback show competitive performance (HumanEval pass@1 of 37.10% for CodeFuse-13B-SFT) and superior handling of Chinese prompts in code generation, translation, comments, and testcase generation. The work emphasizes practical deployment within AntGroup, introduces deployment optimizations (quantization, TensorRT-LLM, caching), and outlines future plans to broaden community resources through MFTCoder, Sparrow, and extended benchmarks.

Abstract

Code Large Language Models (Code LLMs) have gained significant attention in the industry due to their wide applications in the full lifecycle of software engineering. However, the effectiveness of existing models in understanding non-English inputs for multi-lingual code-related tasks is still far from well studied. This paper introduces CodeFuse-13B, an open-sourced pre-trained code LLM. It is specifically designed for code-related tasks with both English and Chinese prompts and supports over 40 programming languages. CodeFuse achieves its effectiveness by utilizing a high quality pre-training dataset that is carefully filtered by program analyzers and optimized during the training process. Extensive experiments are conducted using real-world usage scenarios, the industry-standard benchmark HumanEval-x, and the specially designed CodeFuseEval for Chinese prompts. To assess the effectiveness of CodeFuse, we actively collected valuable human feedback from the AntGroup's software development process where CodeFuse has been successfully deployed. The results demonstrate that CodeFuse-13B achieves a HumanEval pass@1 score of 37.10%, positioning it as one of the top multi-lingual code LLMs with similar parameter sizes. In practical scenarios, such as code generation, code translation, code comments, and testcase generation, CodeFuse performs better than other models when confronted with Chinese prompts.
Paper Structure (20 sections, 1 equation, 6 figures, 7 tables)

This paper contains 20 sections, 1 equation, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Data Preparation
  3. Overview of data processing
  4. Program feature analysis
  5. Code semantic extraction
  6. Code dataset portrait
  7. Training
  8. Tokenization
  9. Model architecture
  10. Pre-training
  11. Supervised Finetuning
  12. Model operations
  13. Evaluation
  14. Evaluation benchmarks and protocols
  15. Compared models
  16. ...and 5 more sections

Figures (6)

  • Figure 1: The diagram of data processing.
  • Figure 2: Distribution of programming languages.
  • Figure 4: Comparison CodeFuse architecture with GPT.
  • Figure 5: An example of ChatML.
  • Figure 6: Examples of CodeFuse generation with prompts in Chinese. The results are generated by CodeFuse VSCode extension, and indicators such as "#" have been removed for better human readability.
  • ...and 1 more figures