Relational Programming with Foundation Models

TL;DR

Vieira presents a declarative relational framework for programming with foundation models, unifying in-context learning, retrieval, and code interpretation through a stateless, plugin-based interface. By extending the Scallop compiler with a foreign-interface, Vieira enables seamless grounding of foundation models as Foreign Predicates and Foreign Attributes, supporting diverse tasks across language, vision, and multi-modal data. Across nine benchmark tasks, Vieira demonstrates concise, readable programs with competitive or superior accuracy compared to baselines without model training, highlighting strong programmability and modular multi-model composition. The work advances practical, interpretable, and scalable use of foundation models in end-to-end AI applications, with open-source tooling and a broad plugin library for ongoing evolution.

Abstract

Foundation models have vast potential to enable diverse AI applications. The powerful yet incomplete nature of these models has spurred a wide range of mechanisms to augment them with capabilities such as in-context learning, information retrieval, and code interpreting. We propose Vieira, a declarative framework that unifies these mechanisms in a general solution for programming with foundation models. Vieira follows a probabilistic relational paradigm and treats foundation models as stateless functions with relational inputs and outputs. It supports neuro-symbolic applications by enabling the seamless combination of such models with logic programs, as well as complex, multi-modal applications by streamlining the composition of diverse sub-models. We implement Vieira by extending the Scallop compiler with a foreign interface that supports foundation models as plugins. We implement plugins for 12 foundation models including GPT, CLIP, and SAM. We evaluate Vieira on 9 challenging tasks that span language, vision, and structured and vector databases. Our evaluation shows that programs in Vieira are concise, can incorporate modern foundation models, and have comparable or better accuracy than competitive baselines.

Figures (12)

• Figure 1: Programs in Vieira using foundation models.
• Figure 2: Snippet of Python implementation of the foreign attribute clip which uses the CLIP model for image classification. Notice that the FA clip returns the FP run_clip.
• Figure 3: Benchmark tasks. The top of each box lists the dataset(s) and the foundation models used in our solutions.
• Figure 4: Illustrative comparisons between our solution and GPT-4 (zero-shot CoT) on selected questions from DR, CLUTRR, and GSM8K datasets. We also include the extracted relations used for subsequent reasoning.
• Figure 5: Systematic generalizability comparisons on the CLUTRR and TSO datasets.