Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bongards at the Boundary of Perception and Reasoning: Programs or Language?

Cassidy Langenfeld, Claas Beger, Gloria Geng, Wasu Top Piriyakulkij, Keya Hu, Yewen Pu, Kevin Ellis

TL;DR

This work addresses visual reasoning limits by evaluating Bongard Problems (BPs) as a testbed for generalization beyond perceptual cues. It introduces a neurosymbolic pipeline that uses Vision-Language Models to hypothesize parameterized rules and Bayesian optimization to fit executable programs, complemented by a verifier that can leverage either programmatic reasoning or Chain-of-Thought prompting. The approach demonstrates that combining natural language reasoning with executable programs yields superior performance on both verifying ground-truth BP rules and solving BPs from scratch, surpassing average human accuracy on the solution task in the reported experiments. The findings suggest a promising direction for AI systems to acquire and operate with new perceptual concepts through integrated NL and symbolic reasoning, with implications for robust concept learning in open-world visual tasks.

Abstract

Vision-Language Models (VLMs) have made great strides in everyday visual tasks, such as captioning a natural image, or answering commonsense questions about such images. But humans possess the puzzling ability to deploy their visual reasoning abilities in radically new situations, a skill rigorously tested by the classic set of visual reasoning challenges known as the Bongard problems. We present a neurosymbolic approach to solving these problems: given a hypothesized solution rule for a Bongard problem, we leverage LLMs to generate parameterized programmatic representations for the rule and perform parameter fitting using Bayesian optimization. We evaluate our method on classifying Bongard problem images given the ground truth rule, as well as on solving the problems from scratch.

Bongards at the Boundary of Perception and Reasoning: Programs or Language?

TL;DR

This work addresses visual reasoning limits by evaluating Bongard Problems (BPs) as a testbed for generalization beyond perceptual cues. It introduces a neurosymbolic pipeline that uses Vision-Language Models to hypothesize parameterized rules and Bayesian optimization to fit executable programs, complemented by a verifier that can leverage either programmatic reasoning or Chain-of-Thought prompting. The approach demonstrates that combining natural language reasoning with executable programs yields superior performance on both verifying ground-truth BP rules and solving BPs from scratch, surpassing average human accuracy on the solution task in the reported experiments. The findings suggest a promising direction for AI systems to acquire and operate with new perceptual concepts through integrated NL and symbolic reasoning, with implications for robust concept learning in open-world visual tasks.

Abstract

Vision-Language Models (VLMs) have made great strides in everyday visual tasks, such as captioning a natural image, or answering commonsense questions about such images. But humans possess the puzzling ability to deploy their visual reasoning abilities in radically new situations, a skill rigorously tested by the classic set of visual reasoning challenges known as the Bongard problems. We present a neurosymbolic approach to solving these problems: given a hypothesized solution rule for a Bongard problem, we leverage LLMs to generate parameterized programmatic representations for the rule and perform parameter fitting using Bayesian optimization. We evaluate our method on classifying Bongard problem images given the ground truth rule, as well as on solving the problems from scratch.
Paper Structure (26 sections, 2 equations, 5 figures, 2 tables)

This paper contains 26 sections, 2 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The Bongard Problems
  3. Related Work
  4. Method
  5. System Overview
  6. Hypothesis Generator
  7. Verifier
  8. Verification with Programs
  9. Parameterized Programs and Optimization
  10. Results
  11. Baselines
  12. Verification Results
  13. Solution Results
  14. Discussion
  15. Conclusion
  16. ...and 11 more sections

Figures (5)

  • Figure 1: (A) Example Bongard problems, each of which consists of 6 positive examples (left drawings) and 6 negative examples. The natural language descriptions shown above each category are not provided to the learner. (B) In comparison, the well-known Raven's Progressive Matrices trade perceptual richness for deeper logical composition.
  • Figure 2: Our system comprises (A) a hypothesis generator that samples possible rules from a VLM and (B) a verifier that combines natural language and code to score and select the best rules.
  • Figure 3: Overview of the program synthesis module of our verifier. Programs are sampled and undergo parameter fitting via Bayesian optimization. If we can successfully optimize programs (i.e., find programs that score at least 0.9 on the training examples), the highest-scoring programs are evaluated on the test examples, with the majority label from these evaluations serving as the output label.
  • Figure 4: Solving 'Elongated' vs. 'Compact' requires a parameter $threshold$ optimized via Bayesian optimization.
  • Figure 5: Examples of problems with similar/dissimilar human/model performance