Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analysis on LLMs Performance for Code Summarization

Md. Ahnaf Akib, Md. Muktadir Mazumder, Salman Ahsan

TL;DR

This study analyzes open-source large language models for code summarization, focusing on LLaMA-3, Phi-3, Mistral, and Gemma using CodeXGLUE benchmarks across Python, Java, Go, JavaScript, PHP, and Ruby. It employs a robust, multi-metric evaluation with BLEU and ROUGE-L and includes fine-tuning considerations to compare model performance across languages. The results show Mistral-7b and Phi-3-medium as the strongest performers overall, with language-dependent strengths and some models excelling in specific datasets. The work provides a practical evaluation framework and guidance for selecting open-source LLMs for code understanding tasks, while highlighting avenues to improve code semantics and model efficiency.

Abstract

Code summarization aims to generate concise natural language descriptions for source code. Deep learning has been used more and more recently in software engineering, particularly for tasks like code creation and summarization. Specifically, it appears that the most current Large Language Models with coding perform well on these tasks. Large Language Models (LLMs) have significantly advanced the field of code summarization, providing sophisticated methods for generating concise and accurate summaries of source code. This study aims to perform a comparative analysis of several open-source LLMs, namely LLaMA-3, Phi-3, Mistral, and Gemma. These models' performance is assessed using important metrics such as BLEU\textsubscript{3.1} and ROUGE\textsubscript{3.2}. Through this analysis, we seek to identify the strengths and weaknesses of each model, offering insights into their applicability and effectiveness in code summarization tasks. Our findings contribute to the ongoing development and refinement of LLMs, supporting their integration into tools that enhance software development and maintenance processes.

Analysis on LLMs Performance for Code Summarization

TL;DR

This study analyzes open-source large language models for code summarization, focusing on LLaMA-3, Phi-3, Mistral, and Gemma using CodeXGLUE benchmarks across Python, Java, Go, JavaScript, PHP, and Ruby. It employs a robust, multi-metric evaluation with BLEU and ROUGE-L and includes fine-tuning considerations to compare model performance across languages. The results show Mistral-7b and Phi-3-medium as the strongest performers overall, with language-dependent strengths and some models excelling in specific datasets. The work provides a practical evaluation framework and guidance for selecting open-source LLMs for code understanding tasks, while highlighting avenues to improve code semantics and model efficiency.

Abstract

Code summarization aims to generate concise natural language descriptions for source code. Deep learning has been used more and more recently in software engineering, particularly for tasks like code creation and summarization. Specifically, it appears that the most current Large Language Models with coding perform well on these tasks. Large Language Models (LLMs) have significantly advanced the field of code summarization, providing sophisticated methods for generating concise and accurate summaries of source code. This study aims to perform a comparative analysis of several open-source LLMs, namely LLaMA-3, Phi-3, Mistral, and Gemma. These models' performance is assessed using important metrics such as BLEU\textsubscript{3.1} and ROUGE\textsubscript{3.2}. Through this analysis, we seek to identify the strengths and weaknesses of each model, offering insights into their applicability and effectiveness in code summarization tasks. Our findings contribute to the ongoing development and refinement of LLMs, supporting their integration into tools that enhance software development and maintenance processes.

Paper Structure

This paper contains 37 sections, 12 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Overview
  3. Motivation and Scope
  4. Problem Statement
  5. Research Challenges
  6. Contribution
  7. Organization
  8. Related Works
  9. RNN Based Models
  10. Tree/GNN Based Models
  11. Transformer Based Models
  12. Large Language Models
  13. Methodology
  14. Proposed Approach
  15. Dataset Collection
  16. ...and 22 more sections

Figures (12)

  • Figure 1: A Simple block of code or function
  • Figure 2: Combining keywords to generate code summaries
  • Figure 3: Illustration of Vanishing and Exploding Gradients in RNNs
  • Figure 4: Preprocessing in GNN
  • Figure 5: Transformer-based Architecture
  • ...and 7 more figures