AdaCoder: Adaptive Prompt Compression for Programmatic Visual Question Answering

TL;DR

AdaCoder tackles the problem of long prompts in visual programmatic models for visual question answering by introducing an adaptive prompt compression framework. It operates in two phases: a compression phase that builds a library of compressed API definitions and question-type-specific code snippets, and an inference phase that classifies questions and assembles a compressed preprompt to generate and execute Python code with a single frozen LLM. The key contributions are: (1) a training-free method that reduces prompt length by about $71.1\%$ while maintaining or improving QA accuracy across multiple VQA datasets, (2) a formalization of compression and inference procedures for code generation with a single frozen LLM, and (3) empirical evidence that adaptive, type-specific code snippets and QA classification are crucial for performance. The approach has practical impact by enabling efficient, scalable VPM deployment with black-box LLMs like GPT and Claude, though it relies on QA-type classification and presents avenues for incorporating white-box models in future work.

Abstract

Visual question answering aims to provide responses to natural language questions given visual input. Recently, visual programmatic models (VPMs), which generate executable programs to answer questions through large language models (LLMs), have attracted research interest. However, they often require long input prompts to provide the LLM with sufficient API usage details to generate relevant code. To address this limitation, we propose AdaCoder, an adaptive prompt compression framework for VPMs. AdaCoder operates in two phases: a compression phase and an inference phase. In the compression phase, given a preprompt that describes all API definitions in the Python language with example snippets of code, a set of compressed preprompts is generated, each depending on a specific question type. In the inference phase, given an input question, AdaCoder predicts the question type and chooses the appropriate corresponding compressed preprompt to generate code to answer the question. Notably, AdaCoder employs a single frozen LLM and pre-defined prompts, negating the necessity of additional training and maintaining adaptability across different powerful black-box LLMs such as GPT and Claude. In experiments, we apply AdaCoder to ViperGPT and demonstrate that it reduces token length by 71.1%, while maintaining or even improving the performance of visual question answering.

Figures (7)

• Figure 1: Inference procedure of AdaCoder. Given an input question, an adaptive programming instruction $\hat{p}_\text{\scriptsize pre}$ for the specific question type is used to generate a Python program for visual question answering.
• Figure 2: AdaCoder framework. (a) Compression phase generates a set of compressed prompts $\mathcal{C}$ by utilizing an LLM $\pi$ with two instructions $r_{\text{\scriptsize pre}}$ for rewriting API definitions and $r_{\text{\scriptsize code}} + r_{\text{\scriptsize sp}}$ for writing snippets of code specialized for each question type $t$. (b) Inference phase adaptively selects code snippets to create compressed preprompt $\hat{p}_\text{\scriptsize pre}$ for generating a Python code $z$ for visual question answering.
• Figure 3: Instruction prompts for the compression phase. {type_definition[i]} is a placeholder to which a question type definition $d_{t}$. See Table \ref{['tab:gqa_qtypes']} for example type definitions.
• Figure 4: Classification prompt for the inference phase. {type[i]} and {type_definition[i]} are placeholders for names and definitions of question type, respectively, for $i = 0, 1, \cdots, n$ where $n$ is the number of question types.
• Figure 5: Confusion matrix of question classification. Overall classification accuracy was 58.1%. (GQA dataset, gpt-3.5-turbo)