Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generation

Yitong Zhang, Chengze Li, Ruize Chen, Guowei Yang, Xiaoran Jia, Yijie Ren, Jia Li

Abstract

Large Language Models (LLMs) have shown strong potential for code generation, yet they remain limited in private-library-oriented code generation, where the goal is to generate code using APIs from private libraries. Existing approaches mainly rely on retrieving private-library API documentation and injecting relevant knowledge into the context at inference time. However, our study shows that this is insufficient: even given accurate required knowledge, LLMs still struggle to invoke private-library APIs effectively. To address this limitation, we propose PriCoder, an approach that teaches LLMs to invoke private-library APIs through automatically synthesized data. Specifically, PriCoder models private-library data synthesis as the construction of a graph, and alternates between two graph operators: (1) Progressive Graph Evolution, which improves data diversity by progressively synthesizing more diverse training samples from basic ones, and (2) Multidimensional Graph Pruning, which improves data quality through a rigorous filtering pipeline. To support rigorous evaluation, we construct two new benchmarks based on recently released libraries that are unfamiliar to the tested models. Experiments on three mainstream LLMs show that PriCoder substantially improves private-library-oriented code generation, yielding gains of over 20% in pass@1 in many settings, while causing negligible impact on general code generation capability. Our code and benchmarks are publicly available at https://github.com/eniacode/PriCoder.

To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generation

Abstract

Large Language Models (LLMs) have shown strong potential for code generation, yet they remain limited in private-library-oriented code generation, where the goal is to generate code using APIs from private libraries. Existing approaches mainly rely on retrieving private-library API documentation and injecting relevant knowledge into the context at inference time. However, our study shows that this is insufficient: even given accurate required knowledge, LLMs still struggle to invoke private-library APIs effectively. To address this limitation, we propose PriCoder, an approach that teaches LLMs to invoke private-library APIs through automatically synthesized data. Specifically, PriCoder models private-library data synthesis as the construction of a graph, and alternates between two graph operators: (1) Progressive Graph Evolution, which improves data diversity by progressively synthesizing more diverse training samples from basic ones, and (2) Multidimensional Graph Pruning, which improves data quality through a rigorous filtering pipeline. To support rigorous evaluation, we construct two new benchmarks based on recently released libraries that are unfamiliar to the tested models. Experiments on three mainstream LLMs show that PriCoder substantially improves private-library-oriented code generation, yielding gains of over 20% in pass@1 in many settings, while causing negligible impact on general code generation capability. Our code and benchmarks are publicly available at https://github.com/eniacode/PriCoder.
Paper Structure (27 sections, 6 equations, 5 figures, 6 tables, 3 algorithms)

This paper contains 27 sections, 6 equations, 5 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Large Language Models for Code Generation
  4. Private-Library-Oriented Code Generation
  5. Motivation
  6. Methodology
  7. Overview
  8. Progressive Graph Evolution
  9. Multidimensional Graph Pruning
  10. Training and Deployment
  11. Benchmark Construction
  12. Experimental Setup
  13. Research Questions
  14. Benchmarks
  15. Metrics
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Two cases on NdonnxEval with Deepseek-Coder-6.7B-Instruct. Despite having access to oracle required API knowledge, the model fails to invoke these APIs effectively.
  • Figure 2: Overview of PriCoder. Starting from API nodes, PriCoder progressively evolves the synthesis graph to construct increasingly diverse sample nodes, while multidimensional graph pruning removes low-quality nodes through syntactic verification, execution verification, and overall functionality verification.
  • Figure 3: Case Study on HumanEval with LLaMa-8B.
  • Figure 4: pass@5 and exec@5 on NdonnxEval with different synthesized training data scale.
  • Figure 5: Impact of the model used for synthesizing training data on the effectiveness of PriCoder.