Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Read between the lines -- Functionality Extraction From READMEs

Prince Kumar, Srikanth Tamilselvam, Dinesh Garg

TL;DR

This paper introduces a novel and useful variant of text summarization called functionality extraction from Git README files, and shows that small size fine-tuned models beat any baseline models that can be designed using popular black-box or white-box large language models such as ChatGPT and Bard.

Abstract

While text summarization is a well-known NLP task, in this paper, we introduce a novel and useful variant of it called functionality extraction from Git README files. Though this task is a text2text generation at an abstract level, it involves its own peculiarities and challenges making existing text2text generation systems not very useful. The motivation behind this task stems from a recent surge in research and development activities around the use of large language models for code-related tasks, such as code refactoring, code summarization, etc. We also release a human-annotated dataset called FuncRead, and develop a battery of models for the task. Our exhaustive experimentation shows that small size fine-tuned models beat any baseline models that can be designed using popular black-box or white-box large language models (LLMs) such as ChatGPT and Bard. Our best fine-tuned 7 Billion CodeLlama model exhibit 70% and 20% gain on the F1 score against ChatGPT and Bard respectively.

Read between the lines -- Functionality Extraction From READMEs

TL;DR

This paper introduces a novel and useful variant of text summarization called functionality extraction from Git README files, and shows that small size fine-tuned models beat any baseline models that can be designed using popular black-box or white-box large language models such as ChatGPT and Bard.

Abstract

While text summarization is a well-known NLP task, in this paper, we introduce a novel and useful variant of it called functionality extraction from Git README files. Though this task is a text2text generation at an abstract level, it involves its own peculiarities and challenges making existing text2text generation systems not very useful. The motivation behind this task stems from a recent surge in research and development activities around the use of large language models for code-related tasks, such as code refactoring, code summarization, etc. We also release a human-annotated dataset called FuncRead, and develop a battery of models for the task. Our exhaustive experimentation shows that small size fine-tuned models beat any baseline models that can be designed using popular black-box or white-box large language models (LLMs) such as ChatGPT and Bard. Our best fine-tuned 7 Billion CodeLlama model exhibit 70% and 20% gain on the F1 score against ChatGPT and Bard respectively.
Paper Structure (20 sections, 4 figures, 17 tables)

This paper contains 20 sections, 4 figures, 17 tables.

Table of Contents

  1. Introduction
  2. FuncRead Dataset
  3. Dataset Collection
  4. Dataset Annotation
  5. Annotation Validation
  6. Task Modelling
  7. Experiments and Results
  8. Evaluation Metrics
  9. Results
  10. Conclusion
  11. Appendix
  12. Limitations
  13. Dataset
  14. Annotators Profile
  15. Annotation Instructions
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Snapshot of Github README content of Daytrader, an online trading application is captured in (a). The human annotated four functionalities based on the description are listed as golden truth along with the functionalities generated by fine-tuned 7 billion CodeLlama model.
  • Figure 2: Ratings distribution of the two annotators during the verification step of the FuncRead dataset.
  • Figure 3: One-to-One bipartite matching (red color) and Many-to-one bipartite matching (blue color). Edges are established based on cosine similarity
  • Figure 4: Functionalities count distribution of the FuncRead dataset.