FETA: Towards Specializing Foundation Models for Expert Task Applications

TL;DR

FETA introduces a first-of-its-kind benchmark and automatic annotation pipeline for evaluating foundation models on expert document understanding tasks, focusing on text-to-image and image-to-text retrieval in car manuals and IKEA catalogs. It adapts CLIP with multiple-instance learning (MIL-CLIP) and MIL-NCE losses to handle many text candidates per image, and demonstrates that standard CLIP struggles on expert-domain data while MIL-based finetuning, especially with LoRA adapters, yields substantial gains. The dataset, consisting of ~56K images and ~89K texts from 349 car-manuals and 26 IKEA catalogs, is designed to be easily extensible to other expert domains, with manual annotations validating automatic metrics. Overall, the work provides practical guidance on fine-tuning FMs for expert tasks and establishes a scalable framework for expanding FM capabilities beyond common-object benchmarks.

Abstract

Foundation Models (FMs) have demonstrated unprecedented capabilities including zero-shot learning, high fidelity data synthesis, and out of domain generalization. However, as we show in this paper, FMs still have poor out-of-the-box performance on expert tasks (e.g. retrieval of car manuals technical illustrations from language queries), data for which is either unseen or belonging to a long-tail part of the data distribution of the huge datasets used for FM pre-training. This underlines the necessity to explicitly evaluate and finetune FMs on such expert tasks, arguably ones that appear the most in practical real-world applications. In this paper, we propose a first of its kind FETA benchmark built around the task of teaching FMs to understand technical documentation, via learning to match their graphical illustrations to corresponding language descriptions. Our FETA benchmark focuses on text-to-image and image-to-text retrieval in public car manuals and sales catalogue brochures. FETA is equipped with a procedure for completely automatic annotation extraction (code would be released upon acceptance), allowing easy extension of FETA to more documentation types and application domains in the future. Our automatic annotation leads to an automated performance metric shown to be consistent with metrics computed on human-curated annotations (also released). We provide multiple baselines and analysis of popular FMs on FETA leading to several interesting findings that we believe would be very valuable to the FM community, paving the way towards real-world application of FMs for practical expert tasks currently 'overlooked' by standard benchmarks focusing on common objects.

Figures (11)

• Figure 1: We introduce FETA, a novel dataset and benchmark for evaluating and improving Foundation (V&L) Models performance on expert data tasks. In contrast to mainstream benchmarks used to evaluate FMs, FETA does not focus on common objects captured with consumer cameras (left), instead providing a completely automatic pipeline for extracting (mostly other visual domains) expert data from publicly available technical and other documentation. Also, as opposed to original CLIP, FETA’s MIL-CLIP method can learn from multiple-hypothesis data automatically extracted from complex document's pages (see right) comprised of multiple images and texts without apparent 1:1 association.
• Figure 2: Simplified figure explaining the automatic annotation process showed using an example page from the cars dataset.
• Figure 3: A Visual representation of the most common nouns in the Car-Manuals and IKEA datasets, compared to COCO, Flickr30K, and CC3M
• Figure 4: A Visual representation of the most common adjectives in the Car-Manuals and IKEA datasets, compared to COCO, Flickr30K, and CC3M
• Figure 5: Illustrative explanation of our MIL variants: An example of one way MIL, from one image to a bag of texts. The figure is added for intuition only, the accurate details are in main paper section 3.