Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards A Generalist Code Embedding Model Based On Massive Data Synthesis

Chaofan Li, Jianlyu Chen, Yingxia Shao, Defu Lian, Zheng Liu

TL;DR

CodeR tackles the scarcity of high-quality training data for generalist code embeddings by constructing CodeR-Pile through a DRU-guided synthetic data pipeline and training a generalist code embedding model with a three-stage Annealing curriculum. The approach yields strong in-domain and out-of-domain code retrieval performance across 16 benchmarks, surpassing existing baselines and closing gaps with proprietary resources. The work provides comprehensive analyses of synthetic data, training dynamics, and ablations, and releases code, models, and data for reproducibility. This has practical implications for improving retrieval-augmented code generation and enabling scalable, domain-general code retrieval systems.

Abstract

Code embedding models attract increasing attention due to the widespread popularity of retrieval-augmented generation (RAG) in software development. These models are expected to capture the rich semantic relationships inherent to code, which differ significantly from those found in text. However, existing models remain severely limited due to the scarcity of high-quality training data. In this work, we introduce \textbf{CodeR} (\underline{Code} \underline{R}etrieval), a state-of-the-art embedding model for general-purpose code retrieval. The superior performance of CodeR is built upon CodeR-Pile, a large-scale synthetic dataset constructed under the DRU (Diversity, Reliability, Usability) principle via a novel data synthesis pipeline. To optimize training effectiveness, we propose Annealing, a curriculum learning strategy that enables effective knowledge transfer across heterogeneous sources of data. We evaluate CodeR based on 16 diverse code retrieval tasks, where it significantly outperforms existing baselines and exhibits strong out-of-domain generalization performance. We have publicly released our code and the well-trained model to facilitate further research in this critical area. https://github.com/FlagOpen/FlagEmbedding/tree/master/research/BGE_Coder.

Towards A Generalist Code Embedding Model Based On Massive Data Synthesis

TL;DR

CodeR tackles the scarcity of high-quality training data for generalist code embeddings by constructing CodeR-Pile through a DRU-guided synthetic data pipeline and training a generalist code embedding model with a three-stage Annealing curriculum. The approach yields strong in-domain and out-of-domain code retrieval performance across 16 benchmarks, surpassing existing baselines and closing gaps with proprietary resources. The work provides comprehensive analyses of synthetic data, training dynamics, and ablations, and releases code, models, and data for reproducibility. This has practical implications for improving retrieval-augmented code generation and enabling scalable, domain-general code retrieval systems.

Abstract

Code embedding models attract increasing attention due to the widespread popularity of retrieval-augmented generation (RAG) in software development. These models are expected to capture the rich semantic relationships inherent to code, which differ significantly from those found in text. However, existing models remain severely limited due to the scarcity of high-quality training data. In this work, we introduce \textbf{CodeR} (\underline{Code} \underline{R}etrieval), a state-of-the-art embedding model for general-purpose code retrieval. The superior performance of CodeR is built upon CodeR-Pile, a large-scale synthetic dataset constructed under the DRU (Diversity, Reliability, Usability) principle via a novel data synthesis pipeline. To optimize training effectiveness, we propose Annealing, a curriculum learning strategy that enables effective knowledge transfer across heterogeneous sources of data. We evaluate CodeR based on 16 diverse code retrieval tasks, where it significantly outperforms existing baselines and exhibits strong out-of-domain generalization performance. We have publicly released our code and the well-trained model to facilitate further research in this critical area. https://github.com/FlagOpen/FlagEmbedding/tree/master/research/BGE_Coder.
Paper Structure (23 sections, 3 equations, 3 figures, 18 tables)

This paper contains 23 sections, 3 equations, 3 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Synthesis
  4. Training Method
  5. Experiment
  6. Benchmarks
  7. Main Results
  8. Analysis of Synthetic Data
  9. Analysis of Training Process
  10. Conclusion
  11. Discussion on Limitations
  12. Overview of Appendix
  13. Instruction Tuning
  14. Experiment Setup
  15. Training Dataset
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Comparison between CodeR-Pile and two existing datasets: CodeSearchNet (CSN) husain2019codesearchnet and CoIR li2024coir. #T: number of code retrieval tasks. #PL: number of programming languages. Size: number of training samples. CoIR (new) refers to the newly introduced data samples in CoIR excluding those already introduced by CodeSearchNet.
  • Figure 2: The data synthesis pipeline of CodeR-Pile.
  • Figure 3: The training pipeline of CodeR.