Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gemini Pro Defeated by GPT-4V: Evidence from Education

Gyeong-Geon Lee, Ehsan Latif, Lehong Shi, Xiaoming Zhai

TL;DR

This paper evaluates GPT-4V and Gemini Pro on multimodal educational tasks, specifically automatic scoring of student-drawn science models using VQA with NERIF prompts. It shows GPT-4V substantially outperforms Gemini Pro in both raw scoring accuracy and agreement metrics, due to superior fine-grained text recognition and retrieval of few-shot exemplars. Despite prompt engineering attempts (NERIF) and input-downsizing, Gemini Pro does not reach GPT-4V's performance, underscoring the importance of intrinsic multimodal capabilities for classroom assessment. The findings have practical implications for selecting AI tools in education and guiding future research on robust multimodal scoring systems.

Abstract

This study compared the classification performance of Gemini Pro and GPT-4V in educational settings. Employing visual question answering (VQA) techniques, the study examined both models' abilities to read text-based rubrics and then automatically score student-drawn models in science education. We employed both quantitative and qualitative analyses using a dataset derived from student-drawn scientific models and employing NERIF (Notation-Enhanced Rubrics for Image Feedback) prompting methods. The findings reveal that GPT-4V significantly outperforms Gemini Pro in terms of scoring accuracy and Quadratic Weighted Kappa. The qualitative analysis reveals that the differences may be due to the models' ability to process fine-grained texts in images and overall image classification performance. Even adapting the NERIF approach by further de-sizing the input images, Gemini Pro seems not able to perform as well as GPT-4V. The findings suggest GPT-4V's superior capability in handling complex multimodal educational tasks. The study concludes that while both models represent advancements in AI, GPT-4V's higher performance makes it a more suitable tool for educational applications involving multimodal data interpretation.

Gemini Pro Defeated by GPT-4V: Evidence from Education

TL;DR

This paper evaluates GPT-4V and Gemini Pro on multimodal educational tasks, specifically automatic scoring of student-drawn science models using VQA with NERIF prompts. It shows GPT-4V substantially outperforms Gemini Pro in both raw scoring accuracy and agreement metrics, due to superior fine-grained text recognition and retrieval of few-shot exemplars. Despite prompt engineering attempts (NERIF) and input-downsizing, Gemini Pro does not reach GPT-4V's performance, underscoring the importance of intrinsic multimodal capabilities for classroom assessment. The findings have practical implications for selecting AI tools in education and guiding future research on robust multimodal scoring systems.

Abstract

This study compared the classification performance of Gemini Pro and GPT-4V in educational settings. Employing visual question answering (VQA) techniques, the study examined both models' abilities to read text-based rubrics and then automatically score student-drawn models in science education. We employed both quantitative and qualitative analyses using a dataset derived from student-drawn scientific models and employing NERIF (Notation-Enhanced Rubrics for Image Feedback) prompting methods. The findings reveal that GPT-4V significantly outperforms Gemini Pro in terms of scoring accuracy and Quadratic Weighted Kappa. The qualitative analysis reveals that the differences may be due to the models' ability to process fine-grained texts in images and overall image classification performance. Even adapting the NERIF approach by further de-sizing the input images, Gemini Pro seems not able to perform as well as GPT-4V. The findings suggest GPT-4V's superior capability in handling complex multimodal educational tasks. The study concludes that while both models represent advancements in AI, GPT-4V's higher performance makes it a more suitable tool for educational applications involving multimodal data interpretation.
Paper Structure (24 sections, 3 equations, 9 figures, 4 tables)

This paper contains 24 sections, 3 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Gemini Pro vs GPT-4V
  3. Architecture
  4. Training Approach
  5. Training Datasets
  6. Performance
  7. Capabilities
  8. Safety
  9. Applications in Education
  10. Methods
  11. Materials and Dataset
  12. Experimental Design
  13. Prompt Design
  14. Findings
  15. Scoring Performance on Student Drawn Models: Gemini Pro vs. GPT-4V
  16. ...and 9 more sections

Figures (9)

  • Figure 1: Architectural Comparison of Gemini Pro and GPT-4V
  • Figure 2: Example Input Image from Task 42
  • Figure 3: Example of Problem Context and Example from Task 42
  • Figure 4: Example Prompt from Task 42
  • Figure 5: Answers from GPT-4V and Gemini Pro When Asked "What Do You See in the Given Image?" (Left: GPT-4V; Right: Gemini Pro)
  • ...and 4 more figures