Transducer Tuning: Efficient Model Adaptation for Software Tasks Using Code Property Graphs

TL;DR

Transducer Tuning addresses the memory and efficiency bottlenecks of fine_tuning large language models for code tasks by leveraging Code Property Graphs. It introduces a modular Transducer with a Graph Vectorization Engine and an Attention_Based Fusion Layer to enrich code embeddings with structural and dependency information while keeping the backbone frozen. Empirical results show up to 99% reduction in trainable parameters with performance competitive to full fine_tuning and on par with/ surpassing other efficient tuning methods across code summarization, assert generation, and code translation. The work demonstrates the practicality of graph_augmented input embeddings for resource_constrained adaptation of large models in software engineering tasks.

Abstract

Large language models have demonstrated promising performance across various software engineering tasks. While fine-tuning is a common practice to adapt these models for downstream tasks, it becomes challenging in resource-constrained environments due to increased memory requirements from growing trainable parameters in increasingly large language models. We introduce \approach, a technique to adapt large models for downstream code tasks using Code Property Graphs (CPGs). Our approach introduces a modular component called \transducer that enriches code embeddings with structural and dependency information from CPGs. The Transducer comprises two key components: Graph Vectorization Engine (GVE) and Attention-Based Fusion Layer (ABFL). GVE extracts CPGs from input source code and transforms them into graph feature vectors. ABFL then fuses those graphs feature vectors with initial code embeddings from a large language model. By optimizing these transducers for different downstream tasks, our approach enhances the models without the need to fine-tune them for specific tasks. We have evaluated \approach on three downstream tasks: code summarization, assert generation, and code translation. Our results demonstrate competitive performance compared to full parameter fine-tuning while reducing up to 99\% trainable parameters to save memory. \approach also remains competitive against other fine-tuning approaches (e.g., LoRA, Prompt-Tuning, Prefix-Tuning) while using only 1.5\%-80\% of their trainable parameters. Our findings show that integrating structural and dependency information through Transducer Tuning enables more efficient model adaptation, making it easier for users to adapt large models in resource-constrained settings.

Figures (4)

• Figure 1: An example of Code Property Graph (CPG). DECL: declaration; PRED: predicate; CALL: function call; Black Edges: syntactic relations; Orange Edges: control flows; Blue Edges: data dependencies; Green Edges: control dependencies.
• Figure 2: The high-level architecture of Transducer Tuning, consisting of two main components: (1) Graph Vectorization Engine for transforming source code into graph feature vectors through Graph Extractor, Graph Vectorizer, and Graph Processing Layer, and (2) Attention-Based Fusion Layer for integrating the graph features with code embeddings from a backbone model to produce enriched code embeddings. The blue colored components are updated during fine-tuning.
• Figure 3: The architecture of the Graph Processing Layer to transform node vectors of CPGs using five components: (1) Normalization for input stabilization, (2) Down Projection for dimensionality reduction, (3) GNN-based Feature Generator for capturing graph relationships, (4) Up Projection for feature reconstruction, and (5) Mean Pooling for generating the final graph representation. The blue colored components are updated during fine-tuning.
• Figure 4: The architecture of the Attention-Based Fusion Layer (ABFL) for integrating graph features with code embeddings using four components: (1) Normalization layers for stabilizing both input embedding and graph feature vector, (2) Attention-Fusion with Query (Q), Key (K), and Value (V) transformations for computing attention weights, and (3) Final Projection for generating the enriched code embedding. The blue colored components are updated during fine-tuning.