Pix2Code: Learning to Compose Neural Visual Concepts as Programs

TL;DR

Pix2Code tackles the challenge of learning abstract visual concepts with minimal supervision by grounding concepts in executable $\lambda$-calculus programs synthesized from symbolic object representations extracted from images. It fuses a neural object extractor with a DreamCoder-inspired wake-sleep program-synthesis loop, producing human-interpretable concept representations that can be revisited and revised with minimal user intervention. The approach demonstrates strong compositional and entity generalization on Kandinsky RelKP and the CURI dataset, and provides interpretable program representations that translate to natural language via LLMs. It further shows potential in real-world settings (MS COCO) and offers a clear revision pathway (XIL) to mitigate confounds and shortcut learning, highlighting practical impact for trustworthy vision systems.

Abstract

The challenge in learning abstract concepts from images in an unsupervised fashion lies in the required integration of visual perception and generalizable relational reasoning. Moreover, the unsupervised nature of this task makes it necessary for human users to be able to understand a model's learnt concepts and potentially revise false behaviours. To tackle both the generalizability and interpretability constraints of visual concept learning, we propose Pix2Code, a framework that extends program synthesis to visual relational reasoning by utilizing the abilities of both explicit, compositional symbolic and implicit neural representations. This is achieved by retrieving object representations from images and synthesizing relational concepts as lambda-calculus programs. We evaluate the diverse properties of Pix2Code on the challenging reasoning domains, Kandinsky Patterns and CURI, thereby testing its ability to identify compositional visual concepts that generalize to novel data and concept configurations. Particularly, in stark contrast to neural approaches, we show that Pix2Code's representations remain human interpretable and can be easily revised for improved performance.

Figures (9)

• Figure 1: Interpretable visual concept learning: learning concepts from few images that can generalize to unseen examples and unseen concepts, such that human users can inspect and potentially revise suboptimal learned concepts.
• Figure 2: The Pix2Code architecture. Objects with bounding box and attribute information are extracted from each positive and negative image example of a visual concept. These representations are converted into a binary classification formulation. The program synthesis component searches for programs to solve each task. This search is based on a probabilistic library that is learned and enhanced during training by frequently used program parts. The result of the search is the visual concept of the image in form of an executable program that can be translated into a corresponding natural language statement.
• Figure 3: Test examples of AllCubes-$5$ (left), AllCubes-$8$ (middle) and AllCubes-$10$ (right) sets. Positive images contain only cubes, while negative images possess all cubes but one cylinder or sphere.
• Figure 4: Class balanced accuracies after revising Pix2Code by removing suboptimal primitives from $L$ on the confounded CURI-Hans set (left) and by adding helpful primitives to $L$ in the counting split of CURI (right). Pix2Code+XIL indicates the revised models.
• Figure 5: Examples of the concept "Exists person and exists dog" based on the COCO dataset.