Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Not All Tokens Matter: Data-Centric Optimization for Efficient Code Summarization

Saima Afrin, Zaiyu Cheng, Tushar Sharma, Alexander Serebrenik, Massimiliano Di Penta, Antonio Mastropaolo

TL;DR

The paper tackles data-centric optimization for code summarization, showing that token-level input reductions and language-specific semantic alignment can dramatically cut training data and compute while maintaining or improving summary quality. It introduces three token-reduction techniques (AST-based input, function-signature extraction, CrystalBLEU-guided pruning) and demonstrates their language-dependent effects across Java and Python, with notable findings that Python benefits from function-signature compression while Java favors structural representations like AST. A Python-specific semantic alignment metric, SIDE_{py}, is developed to align automatic evaluation with human judgments, and its effectiveness is validated alongside robust cross-language experiments, including a new Python benchmark PyBench. The work emphasizes data quality and language-aware strategies as practical levers for efficient and trustworthy code summarization systems, offering a concrete, replication-ready framework for data-centric optimization.

Abstract

Instruction-tuned Language Models ILMs have become essential components of modern AI systems, demonstrating exceptional versatility across a wide range of natural language and reasoning tasks. Among their most impactful applications is code generation, where ILMs--commonly referred to as Code Language Models CLMs--have demonstrated remarkable capability. This strength stems from their defining feature: the use of explicit task instructions during fine-tuning, which enables them to bridge natural language and code by translating human intent into executable code. While much of their progress has been driven by advances in scaling laws and training methodologies, one critical aspect remains underexplored--the impact of system prompts on the performance of both general-purpose ILMs and specialized CLMs when instantiated to assist users with code generation activities. In this study, we take a first step toward bridging this gap by systematically evaluating how system prompts of varying instructional detail, along with model scale, prompting strategy, and programming language, affect ILMs and CLMs in code generation tasks. Our evaluation framework, spanning 120 model configurations, reveals that (1) the influence of system prompts increases with model scale; (2) few-shot prompting reduces this effect compared to zero-shot; and (3) programming language matters, with Java showing greater sensitivity to system prompt variations than Python.

Not All Tokens Matter: Data-Centric Optimization for Efficient Code Summarization

TL;DR

The paper tackles data-centric optimization for code summarization, showing that token-level input reductions and language-specific semantic alignment can dramatically cut training data and compute while maintaining or improving summary quality. It introduces three token-reduction techniques (AST-based input, function-signature extraction, CrystalBLEU-guided pruning) and demonstrates their language-dependent effects across Java and Python, with notable findings that Python benefits from function-signature compression while Java favors structural representations like AST. A Python-specific semantic alignment metric, SIDE_{py}, is developed to align automatic evaluation with human judgments, and its effectiveness is validated alongside robust cross-language experiments, including a new Python benchmark PyBench. The work emphasizes data quality and language-aware strategies as practical levers for efficient and trustworthy code summarization systems, offering a concrete, replication-ready framework for data-centric optimization.

Abstract

Instruction-tuned Language Models ILMs have become essential components of modern AI systems, demonstrating exceptional versatility across a wide range of natural language and reasoning tasks. Among their most impactful applications is code generation, where ILMs--commonly referred to as Code Language Models CLMs--have demonstrated remarkable capability. This strength stems from their defining feature: the use of explicit task instructions during fine-tuning, which enables them to bridge natural language and code by translating human intent into executable code. While much of their progress has been driven by advances in scaling laws and training methodologies, one critical aspect remains underexplored--the impact of system prompts on the performance of both general-purpose ILMs and specialized CLMs when instantiated to assist users with code generation activities. In this study, we take a first step toward bridging this gap by systematically evaluating how system prompts of varying instructional detail, along with model scale, prompting strategy, and programming language, affect ILMs and CLMs in code generation tasks. Our evaluation framework, spanning 120 model configurations, reveals that (1) the influence of system prompts increases with model scale; (2) few-shot prompting reduces this effect compared to zero-shot; and (3) programming language matters, with Java showing greater sensitivity to system prompt variations than Python.
Paper Structure (33 sections, 1 equation, 3 figures, 10 tables)

This paper contains 33 sections, 1 equation, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Lexical-based Evaluation Metrics
  4. Semantic-based Evaluation Metrics
  5. Optimizing Code Summarization Training Dataset via Coherence-Based Filtering
  6. Token-Based Data Optimization
  7. Optimizing Code Summarization Training Dataset via Token-Level Optimization Techniques
  8. Study overview
  9. Study 1: Optimizing Code Summarization Training Dataset via Token-based Reduction Methods
  10. Dataset Optimization
  11. Model Training
  12. Experiments and Evaluation
  13. Evaluation Dataset:
  14. Evaluation Metrics:
  15. Results:
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Different Token Representation Techniques in Token-based Optimization and their computed Shannon entropy value
  • Figure 2: Qualitative comparison of token optimization strategies on Python code summarization. Function Signature extraction (✓) successfully captures the core semantics, generating an accurate summary, while full code tokens, CrystalBLEU, and AST representation (✗) all produce irrelevant summaries that misidentify the function's purpose.
  • Figure 3: Qualitative comparison of token optimization strategies on Java code summarization. Full code tokens (✓) and AST representation (✓) successfully capture the core functionality, while Function Signature and CrystalBLEU (✗) produce incorrect or incomplete summaries.