Training with Pseudo-Code for Instruction Following
Prince Kumar, Rudra Murthy, Riyaz Bhat, Danish Contractor
TL;DR
This work tackles the difficulty of instruction-following in large language models by fine-tuning models with instruction data augmented by pseudo-code re-expressions of instructions. It introduces a multi-stage data generation pipeline that generates pseudo-code, evaluates its correctness, and repairs it when needed, using a Tuluv2-based training mixture and additional datasets. Across 11 public benchmarks and five base models, the approach yields relative gains in instruction-following of up to 19% and an average improvement of up to 14% across tasks, while largely preserving math and common-sense reasoning capabilities; code-task performance can degrade if code-focused data is overrepresented. The findings suggest that pseudo-code instruction representations can serve as a drop-in augmentation to NL prompts, offering a scalable and accessible path to more robust instruction-following, with future directions including optimized code-task mixing, NL and NL-PC blends, and integration with advanced control-flow frameworks.
Abstract
Despite the rapid progress in the capabilities of Large Language Models (LLMs), they continue to have difficulty following relatively simple, unambiguous instructions, especially when compositions are involved. In this paper, we take inspiration from recent work that suggests that models may follow instructions better when they are expressed in pseudo-code. However, writing pseudo-code programs can be tedious and using few-shot demonstrations to craft code representations for use in inference can be unnatural for non-expert users of LLMs. To overcome these limitations, we propose fine-tuning LLMs with instruction-tuning data that additionally includes instructions re-expressed in pseudo-code along with the final response. We evaluate models trained using our method on $11$ publicly available benchmarks comprising of tasks related to instruction-following, mathematics, and common-sense reasoning. We conduct rigorous experiments with $5$ different models and find that not only do models follow instructions better when trained with pseudo-code, they also retain their capabilities on the other tasks related to mathematical and common sense reasoning. Specifically, we observe a relative gain of $3$--$19$% on instruction-following benchmark, and an average gain of upto 14% across all tasks.