Advancing Language Models for Code-related Tasks
Zhao Tian
TL;DR
The paper tackles the gap between strong language models and practical code-related tasks by addressing data quality, architectural representation, and reasoning. It proposes a tripartite framework: CODA and CodeDenoise to improve code data quality, LEAM and LEAM++ for syntax-guided, AST-based code modeling, and muFiX and Specine to enhance model reasoning and requirement alignment. Empirical results show CODA and CodeDenoise yield robustness and accuracy gains, LEAM/LEAM++ achieve syntactic perfection and realistic fault generation, and muFiX and Specine deliver substantial improvements in Pass@1. Together these techniques push code LMs toward more reliable, correct, and requirement-aligned software engineering assistance.
Abstract
Recent advances in language models (LMs) have driven significant progress in various software engineering tasks. However, existing LMs still struggle with complex programming scenarios due to limitations in data quality, model architecture, and reasoning capability. This research systematically addresses these challenges through three complementary directions: (1) improving code data quality with a code difference-guided adversarial augmentation technique (CODA) and a code denoising technique (CodeDenoise); (2) enhancing model architecture via syntax-guided code LMs (LEAM and LEAM++); and (3) advancing model reasoning with a prompting technique (muFiX) and an agent-based technique (Specine). These techniques aim to promote the practical adoption of LMs in software development and further advance intelligent software engineering.