Pyramid Coder: Hierarchical Code Generator for Compositional Visual Question Answering

TL;DR

PyramidCoder targets compositional visual question answering by reframing the problem as a hierarchical code-generation task. It uses a single frozen LLM with three prompts—query rephraser, code generator, and answer aggregator—to produce diverse candidate codes and select the best answer, without any model training. Across GQA, VQAv2, and NLVR2, PyramidCoder achieves consistent accuracy gains over the state-of-the-art CodeVQA, demonstrating robustness across LLMs and datasets. The approach enhances interpretability and flexibility in PVQA by enabling multiple reasoning paths and selective code execution, with practical impact on how flexible QA systems can leverage LLMs for visual reasoning.

Abstract

Visual question answering (VQA) is the task of providing accurate answers to natural language questions based on visual input. Programmatic VQA (PVQA) models have been gaining attention recently. These use large language models (LLMs) to formulate executable programs that address questions requiring complex visual reasoning. However, there are challenges in enabling LLMs to comprehend the usage of image processing modules and generate relevant code. To overcome these challenges, this paper introduces PyramidCoder, a novel prompting framework for PVQA models. PyramidCoder consists of three hierarchical levels, each serving a distinct purpose: query rephrasing, code generation, and answer aggregation. Notably, PyramidCoder utilizes a single frozen LLM and pre-defined prompts at each level, eliminating the need for additional training and ensuring flexibility across various LLM architectures. Compared to the state-of-the-art PVQA model, our approach improves accuracy by at least 0.5% on the GQA dataset, 1.4% on the VQAv2 dataset, and 2.9% on the NLVR2 dataset.

Figures (6)

• Figure 1: PyramidCoder is a hierarchical framework comprising three modules. It generates multiple code candidates, maximizing the diversity of solutions available for a given input query to produce the correct answer effectively.
• Figure 2: Framework overview. (a) PyramidCoder $\Pi$ takes an image $x$ and a query $q$ as input, producing both the code and answer for the input image-query pair. (b) PyramidCoder contains three modules: a query rephraser $R$, a code generator $G$, and an answer aggregator $A$. Image processing APIs are also used in the code-executing process. (c) For each input query, the PyramidCoder generates diverse expressions $r_{i}$ of the input query and multiple code candidates $z_{ij}$, each representing a distinct solution to the given query. Finally, these codes $z_{ij}$ and their corresponding pre-execution results $a_{ij}$ are aggregated to derive the final answer $\hat{a}$ and the accompanying interpretable code $z$.
• Figure 3: Prompt definitions.
• Figure 4: API reference $p_{\text{api}}$.
• Figure 5: QA Accuracy by question type.