Harnessing GPT-4V(ision) for Insurance: A Preliminary Exploration

TL;DR

This work investigates how GPT-4V, a large multimodal model, performs across diverse insurance tasks that combine visual and textual data, spanning auto, property, health, and agricultural domains. It introduces a task-classification framework by insurance type and stage, and employs a qualitative testing pipeline with tailored prompts to assess underwriting, risk assessment, risk monitoring, and claims processing. Key findings show robust multimodal understanding and domain knowledge in many tasks, but reveal notable gaps in precise damage/loss estimation, hallucinations in image interpretation, and multilingual document extraction. The study provides a foundation for integrating LMMs into insurance workflows while outlining concrete challenges that guide future research and practical deployment.

Abstract

The emergence of Large Multimodal Models (LMMs) marks a significant milestone in the development of artificial intelligence. Insurance, as a vast and complex discipline, involves a wide variety of data forms in its operational processes, including text, images, and videos, thereby giving rise to diverse multimodal tasks. Despite this, there has been limited systematic exploration of multimodal tasks specific to insurance, nor a thorough investigation into how LMMs can address these challenges. In this paper, we explore GPT-4V's capabilities in the insurance domain. We categorize multimodal tasks by focusing primarily on visual aspects based on types of insurance (e.g., auto, household/commercial property, health, and agricultural insurance) and insurance stages (e.g., risk assessment, risk monitoring, and claims processing). Our experiment reveals that GPT-4V exhibits remarkable abilities in insurance-related tasks, demonstrating not only a robust understanding of multimodal content in the insurance domain but also a comprehensive knowledge of insurance scenarios. However, there are notable shortcomings: GPT-4V struggles with detailed risk rating and loss assessment, suffers from hallucination in image understanding, and shows variable support for different languages. Through this work, we aim to bridge the insurance domain with cutting-edge LMM technology, facilitate interdisciplinary exchange and development, and provide a foundation for the continued advancement and evolution of future research endeavors.

Figures (40)

• Figure 1: An overview of the task classification in the insurance domain.
• Figure 2: An overview of our approach in exploring GPT-4V(ision).
• Figure 3: The test case of GPT-4V's capability in vehicle odometer reading. The correct answer parts, incorrect answer parts, and references are highlighted in green, red and yellow, respectively.
• Figure 4: The test case of GPT-4V's capability in vehicle underwriting. The relevant parts are highlighted in green.
• Figure 5: The test case of GPT-4V's capability in dangerous driving behavior detection through inward-facing camera. The relevant parts are highlighted in green.