Framework-Aware Code Generation with API Knowledge Graph-Constructed Data: A Study on HarmonyOS
Mingwei Liu, Zheng Pei, Yanlin Wang, Zihao Wang, Zikang Li, Enci Lin, Xin Peng, Zibin Zheng
TL;DR
This work tackles the challenge of generating correct code for low-resource frameworks like HarmonyOS by exposing LLMs to API usage patterns through APIKG4SYN, a knowledge-graph-guided data synthesis framework. It builds a HarmonyOS API knowledge graph, applies uncertainty estimation and Monte Carlo Tree Search to create diverse single-API and multi-API training scenarios, and synthesizes API-oriented question–code tuples for fine-tuning. Empirical results on the HarmonyOS OHBen benchmark show that fine-tuning with APIKG4SYN data significantly improves pass@1 (e.g., up to 25.00% for Qwen2.5-Coder-7B) and underscores the importance of multi-API data and larger data scales in low-resource settings. The work provides open-source data and tools, enabling practical adaptation of smaller models to proprietary or niche frameworks without executable code, with broad implications for enterprise code generation in low-resource domains.
Abstract
In the context of software frameworks with limited resources (such as HarmonyOS), large language models (LLMs) often exhibit poor code generation performance because they lack sufficient exposure to such environments during pre-training. Although LLMs can usually maintain correct logical structures across programming languages, they frequently struggle when dealing with framework-specific APIs or syntax, resulting in errors. This indicates that while pre-training equips LLMs with general algorithmic capabilities, they remain unfamiliar with the distinctive syntax and API usage of underrepresented frameworks. As a result, even advanced commercial models like GPT-4o cannot reliably generate correct code without prior adaptation. To address this issue, we propose APIKG4SYN, a framework designed to exploit API knowledge graphs for the construction of API-oriented question-code pairs, specifically tailored for low-resource frameworks without requiring executable code. APIKG4SYN integrates both single-API and multi-API knowledge, where the latter is derived through uncertainty estimation (UE)-driven Monte Carlo Tree Search (MCTS), enabling the creation of a diverse and informative dataset for fine-tuning LLMs. Using HarmonyOS as a case study, we build the first benchmark for HarmonyOS code generation. Experimental results show that fine-tuning Qwen with APIKG4SYN raises pass@1 accuracy to 25.00%, compared with 17.59% for the baseline GPT model. These results confirm that API-oriented data significantly enhance LLM performance in low-resource software development scenarios.