Can humans teach machines to code?
Céline Hocquette, Johannes Langer, Andrew Cropper, Ute Schmid
TL;DR
This work investigates whether humans can effectively teach machines to code via example-based specifications in inductive program synthesis. Through an empirical study with six list manipulation concepts and five diverse learning systems, the authors compare examples provided by non-experts, an expert, and randomly generated data. They find that non-expert humans generally fail to provide sufficient examples for generalization, while an expert can produce a small, highly informative set of examples that yields perfect accuracy; surprisingly, random synthetic examples often outperform human-provided ones. The results challenge the prevailing assumption that human-provided examples suffice for learning programs and highlight the need for systems that adapt to human inputs or leverage synthetic data that better mimics human teaching signals, with implications for end-user programming and robot-assisted coding.
Abstract
The goal of inductive program synthesis is for a machine to automatically generate a program from user-supplied examples. A key underlying assumption is that humans can provide sufficient examples to teach a concept to a machine. To evaluate the validity of this assumption, we conduct a study where human participants provide examples for six programming concepts, such as finding the maximum element of a list. We evaluate the generalisation performance of five program synthesis systems trained on input-output examples (i) from non-expert humans, (ii) from a human expert, and (iii) randomly sampled. Our results suggest that non-experts typically do not provide sufficient examples for a program synthesis system to learn an accurate program.