Synthetic Function Demonstrations Improve Generation in Low-Resource Programming Languages

TL;DR

The paper tackles generating outputs in low-resource programming languages by synthesizing textbook-style Excel function demonstrations from public documentation and finetuning models on this curriculum. It shows that finetuning on synthetic QA data yields meaningful improvements in table-based QA performance, often surpassing standard RAG approaches, and that prior HRPL pretraining facilitates adaptation to LRPL tasks. The approach is validated across two open-model families (Qwen 2.5 and Llama 2) and two Excel-recast QA datasets (WikiTQ and TAT-QA), with notable gains attributed to synthetic data quality and alignment. The findings suggest a scalable path to extend LLM capabilities to long-tail programming domains and encourage broader use of synthetic data in pretraining for domain adaptation.

Abstract

A key consideration when training an LLM is whether the target language is more or less resourced, whether this is English compared to Welsh, or Python compared to Excel. Typical training data for programming languages consist of real program demonstrations coupled with human-written comments. Here we present novel approaches to the creation of such data for low resource programming languages. We generate fully-synthetic, textbook-quality demonstrations of common library functions in an example domain of Excel formulas, using a teacher model. We then finetune an underperforming student model, and show improvement on 2 question-answering datasets recast into the Excel domain. We show advantages of finetuning over standard, off-the-shelf RAG approaches, which can offer only modest improvement due to the unfamiliar target domain.

Figures (6)

• Figure 1: Finetuning on synthetic data improves adaptation to the low-resource domain. Qwen2.5-coder 3B learns to read the table and compose a formula with flexible string formatting, executing to the answer.
• Figure 2: We match a table to each function by querying GPT-4o. First, we instruct the model to generate reasoning for each table about how it could be suitable, then we instruct it to choose the table with the best reasoning.
• Figure 3: We instruct the teacher model GPT-4o to generate multiple tutorials for a given function, demonstrating each function argument. The generation is in JSON, which is then compiled into corresponding textbook demonstrations for the student model using a simple Jinja template. See Figure \ref{['fig:demo_generated_examples']} for a sample JSON output, and Figure \ref{['fig:demo_compiled_tutorial']} for a compiled tutorial.
• Figure 4: One query to the teacher model for the MATCH function yields three examples formatted as JSON. We compile each example into plain text using a Jinja template, for training the student with a simple next-token prediction objective. See Figure \ref{['fig:demo_compiled_tutorial']} for an example tutorial.
• Figure 5: A textbook-quality demonstration of how to use the MATCH function to find the position of the string "Chile" in a list, using an exact match. The teacher's chain of thought reasoning is included to explain how to gather information from the table and construct the final formula.