Programming Not Only by Example
Hila Peleg, Sharon Shoham, Eran Yahav
TL;DR
The paper addresses the challenge of conveying programmer intent in program synthesis, especially for end-users and professionals solving small tasks. It introduces the Granular Interaction Model (GIM), which augments synthesized programs with debug information and enables granular, code-level feedback (remove/retain/affix) in addition to traditional input-output examples. The authors formalize the insufficiency of examples alone to prune undesired constructs and provide a formal framework for granular predicates, a debugging view, and a practical synthesizer. Through ideal-synthesis experiments and a controlled user study with 32 developers, they show that granular feedback is faster to generate, preferred by users, and can significantly reduce search space, though examples still play a beneficial role, suggesting a hybrid approach.
Abstract
In recent years, there has been tremendous progress in automated synthesis techniques that are able to automatically generate code based on some intent expressed by the programmer. A major challenge for the adoption of synthesis remains in having the programmer communicate their intent. When the expressed intent is coarse-grained (for example, restriction on the expected type of an expression), the synthesizer often produces a long list of results for the programmer to choose from, shifting the heavy-lifting to the user. An alternative approach, successfully used in end-user synthesis is programming by example (PBE), where the user leverages examples to interactively and iteratively refine the intent. However, using only examples is not expressive enough for programmers, who can observe the generated program and refine the intent by directly relating to parts of the generated program. We present a novel approach to interacting with a synthesizer using a granular interaction model. Our approach employs a rich interaction model where (i) the synthesizer decorates a candidate program with debug information that assists in understanding the program and identifying good or bad parts, and (ii) the user is allowed to provide feedback not only on the expected output of a program, but also on the underlying program itself. That is, when the user identifies a program as (partially) correct or incorrect, they can also explicitly indicate the good or bad parts, to allow the synthesizer to accept or discard parts of the program instead of discarding the program as a whole. We show the value of our approach in a controlled user study. Our study shows that participants have strong preference to using granular feedback instead of examples, and are able to provide granular feedback much faster.