Benchmarking Foundation Models on Exceptional Cases: Dataset Creation and Validation

TL;DR

This work introduces the notion of exceptional cases as out-of-distribution reasoning tasks for foundation models and builds a multimodal benchmark spanning graphic novels, calligraphy, lyrics, and real-vs-fake news. Using GPT-4o, Gemini-1.5-Pro, and Claude-3.5-Sonnet, it evaluates zero-shot, CoT, and CoT+Few-Shot prompting across four datasets with tasks including image ordering, OCR, infilling, and description generation, highlighting modality-specific strengths and weaknesses. A key contribution is demonstrating how prompt engineering can mitigate OOD reasoning gaps, while also revealing persistent limitations in language and multimodal understanding; the work also provides supplementary materials and a public codebase at GitHub. The study lays groundwork for future, broader benchmarks that incorporate audio and additional languages, guiding development of more robust, human-like reasoning in FMs.

Abstract

Foundation models (FMs) have achieved significant success across various tasks, leading to research on benchmarks for reasoning abilities. However, there is a lack of studies on FMs performance in exceptional scenarios, which we define as out-of-distribution (OOD) reasoning tasks. This paper is the first to address these cases, developing a novel dataset for evaluation of FMs across multiple modalities, including graphic novels, calligraphy, news articles, and lyrics. It includes tasks for instance classification, character recognition, token prediction, and text generation. The paper also proposes prompt engineering techniques like Chain-of-Thought (CoT) and CoT+Few-Shot to enhance performance. Validation of FMs using various methods revealed improvements. The code repository is accessible at: https://github.com/MLAI-Yonsei/ExceptionalBenchmark

Figures (13)

• Figure 1: Distribution of Exceptional Cases Dataset and summary of four distinct datasets and their subsections.
• Figure 2: We remove clue-containing sections marked by red boxes that help determine the correct storyline. These sections were removed as shown by the blue dotted line boxes in the 'Shuffled and Edited' version.
• Figure 3: Description of the random shuffle experiment process: In the 'Prompt', all essential information is provided, including the fact that all images are shuffled, that the four images are parts of a story, and the response format. The 'Code' section illustrates the task sequence from (a) to (e). (a) shows the shuffled input image order, (b) is the index of the input image order, (c) is GPT-4o's response which is the inferred result, (d) is the transformation from index order to image order, and (e) is the ground truth order used to calculate accuracy.
• Figure 4: Example of the random shuffle task. The original sequence is [1, 4, 3, 2], but GPT-4o produce an incorrect result.
• Figure 5: Correct Order Check: This example shows that while GPT-4o can correctly order the images, it sometimes lacks in scene description such as using mismatched verbs (highlighted in red).