Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Codellm-Devkit: A Framework for Contextualizing Code LLMs with Program Analysis Insights

Rahul Krishna, Rangeet Pan, Raju Pavuluri, Srikanth Tamilselvam, Maja Vukovic, Saurabh Sinha

TL;DR

codellm-devkit (hereafter, cldk), an open-source library that significantly simplifies the process of performing program analysis at various levels of granularity for different programming languages to support code LLM use cases is presented.

Abstract

Large Language Models for Code (or code LLMs) are increasingly gaining popularity and capabilities, offering a wide array of functionalities such as code completion, code generation, code summarization, test generation, code translation, and more. To leverage code LLMs to their full potential, developers must provide code-specific contextual information to the models. These are typically derived and distilled using program analysis tools. However, there exists a significant gap--these static analysis tools are often language-specific and come with a steep learning curve, making their effective use challenging. These tools are tailored to specific program languages, requiring developers to learn and manage multiple tools to cover various aspects of the their code base. Moreover, the complexity of configuring and integrating these tools into the existing development environments add an additional layer of difficulty. This challenge limits the potential benefits that could be gained from more widespread and effective use of static analysis in conjunction with LLMs. To address this challenge, we present codellm-devkit (hereafter, `CLDK'), an open-source library that significantly simplifies the process of performing program analysis at various levels of granularity for different programming languages to support code LLM use cases. As a Python library, CLDK offers developers an intuitive and user-friendly interface, making it incredibly easy to provide rich program analysis context to code LLMs. With this library, developers can effortlessly integrate detailed, code-specific insights that enhance the operational efficiency and effectiveness of LLMs in coding tasks. CLDK is available as an open-source library at https://github.com/IBM/codellm-devkit.

Codellm-Devkit: A Framework for Contextualizing Code LLMs with Program Analysis Insights

TL;DR

codellm-devkit (hereafter, cldk), an open-source library that significantly simplifies the process of performing program analysis at various levels of granularity for different programming languages to support code LLM use cases is presented.

Abstract

Large Language Models for Code (or code LLMs) are increasingly gaining popularity and capabilities, offering a wide array of functionalities such as code completion, code generation, code summarization, test generation, code translation, and more. To leverage code LLMs to their full potential, developers must provide code-specific contextual information to the models. These are typically derived and distilled using program analysis tools. However, there exists a significant gap--these static analysis tools are often language-specific and come with a steep learning curve, making their effective use challenging. These tools are tailored to specific program languages, requiring developers to learn and manage multiple tools to cover various aspects of the their code base. Moreover, the complexity of configuring and integrating these tools into the existing development environments add an additional layer of difficulty. This challenge limits the potential benefits that could be gained from more widespread and effective use of static analysis in conjunction with LLMs. To address this challenge, we present codellm-devkit (hereafter, `CLDK'), an open-source library that significantly simplifies the process of performing program analysis at various levels of granularity for different programming languages to support code LLM use cases. As a Python library, CLDK offers developers an intuitive and user-friendly interface, making it incredibly easy to provide rich program analysis context to code LLMs. With this library, developers can effortlessly integrate detailed, code-specific insights that enhance the operational efficiency and effectiveness of LLMs in coding tasks. CLDK is available as an open-source library at https://github.com/IBM/codellm-devkit.

Paper Structure

This paper contains 28 sections, 13 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Developer Survey on Tool Needs
  4. Interview results
  5. Background of solutions with code LLM
  6. Need for Better/Easier Program Analysis Tools
  7. Support for prompting and other LLM features
  8. Academic Survey on Tool Needs
  9. Dataset collection
  10. Dataset insights
  11. Design Principles
  12. Architectural Overview
  13. CLDK core
  14. Program analysis Engines
  15. Schema
  16. ...and 13 more sections

Figures (13)

  • Figure 1: Landscape of LLMs based solutions for enterprise software systems.
  • Figure 2: CodeQL requires numerous file i/o, subprocess calls, and string split/strip to obtain an output.
  • Figure 3: Tree sitter has a different grammars for similar abstractions in several languages
  • Figure 4: cldk standarizes all API calls. The user only changes the language parameter
  • Figure 6: LLM-assisted SE tasks where program analysis has been an integral part.
  • ...and 8 more figures