GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?

TL;DR

The paper conducts a large-scale, quantitative evaluation of GPT-4’s zero-shot visual recognition capabilities across images, videos, and point clouds, leveraging GPT-4-generated descriptive prompts to augment CLIP-based recognition and using GPT-4V for direct visual predictions. It analyzes 16 benchmarks and shows that linguistic enrichment yields about a 7% average Top-1 gain, while GPT-4V delivers competitive performance relative to EVA-CLIP, with particular strength in video datasets. The study highlights the importance of prompt design, reveals temporal modeling limitations in current GPT-4V for video tasks, and provides practical baselines, ablations, and dataset prompts to guide future multimodal research. It also discusses operational considerations, including testing protocols and budget, offering a valuable data point for the design of future zero-shot vision-language systems.

Abstract

This paper does not present a novel method. Instead, it delves into an essential, yet must-know baseline in light of the latest advancements in Generative Artificial Intelligence (GenAI): the utilization of GPT-4 for visual understanding. Our study centers on the evaluation of GPT-4's linguistic and visual capabilities in zero-shot visual recognition tasks: Firstly, we explore the potential of its generated rich textual descriptions across various categories to enhance recognition performance without any training. Secondly, we evaluate GPT-4's visual proficiency in directly recognizing diverse visual content. We conducted extensive experiments to systematically evaluate GPT-4's performance across images, videos, and point clouds, using 16 benchmark datasets to measure top-1 and top-5 accuracy. Our findings show that GPT-4, enhanced with rich linguistic descriptions, significantly improves zero-shot recognition, offering an average top-1 accuracy increase of 7% across all datasets. GPT-4 excels in visual recognition, outshining OpenAI-CLIP's ViT-L and rivaling EVA-CLIP's ViT-E, particularly in video datasets HMDB-51 and UCF-101, where it leads by 22% and 9%, respectively. We hope this research contributes valuable data points and experience for future studies. We release our code at https://github.com/whwu95/GPT4Vis.

Figures (9)

• Figure 1: An overview of 16 evaluated popular benchmark datasets, comprising images, videos, and point clouds. The image benchmarks include tasks such as texture recognition, satellite image classification, scene recognition, facial expression recognition, as well as fine-grained object classification. The video datasets encompass diverse human actions captured from various viewpoints and scenes. The point cloud datasets provide valuable information that can be projected onto multi-view depth maps for visual recognition.
• Figure 2: Zero-shot visual recognition leveraging GPT-4's linguistic and visual capabilities. (a) We built upon the visual-language bridge established by CLIP clip and employed the rich linguistic knowledge of GPT-4 to generate additional descriptions for categories, exploring the benefits to visual recognition. (b) We present visual content (i.e., single or multiple images) along with a category list, and prompt GPT-4V to generate the top-5 prediction results.
• Figure 3: Processing video and point cloud data into images.
• Figure 4: Sentences generated by GPT-4 for "British Shorthairs".
• Figure 5: Prompts for image, video, and point cloud datasets: (a) An example from RAF-DB li2017reliable illustrates 7-class facial expression recognition. (b) A video example from HMDB-51 hmdb demonstrates 51-class action recognition, where ellipses indicate category names omitted due to space constraints. (c) An example from ModelNet10 modelnet40 for point cloud classification across 10 categories, where ellipses again indicate the truncation of category names owing to space constraints. Please zoom in for best view.