Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CameraBench: Benchmarking Visual Reasoning in MLLMs via Photography

I-Sheng Fang, Jun-Cheng Chen

TL;DR

This paper addresses visual reasoning in photography by introducing CameraBench, a benchmark that tests multimodal LLMs on inferring numerical camera settings from images. It builds CameraSettings25K, normalizes settings to 35mm full-frame, and defines binary-choice and five-choice tasks to probe distinctions across focal length, aperture, ISO, and exposure time. Results show that larger, reasoning-focused MLLMs perform better on certain tasks, but many models struggle with single-parameter distinctions, revealing gaps in visual reasoning for photography. The work underscores the need for physics-aware visual reasoning in practical photo assistant applications and lays groundwork for broader benchmarking across MLLMs.

Abstract

Large language models (LLMs) and multimodal large language models (MLLMs) have significantly advanced artificial intelligence. However, visual reasoning, reasoning involving both visual and textual inputs, remains underexplored. Recent advancements, including the reasoning models like OpenAI o1 and Gemini 2.0 Flash Thinking, which incorporate image inputs, have opened this capability. In this ongoing work, we focus specifically on photography-related tasks because a photo is a visual snapshot of the physical world where the underlying physics (i.e., illumination, blur extent, etc.) interplay with the camera parameters. Successfully reasoning from the visual information of a photo to identify these numerical camera settings requires the MLLMs to have a deeper understanding of the underlying physics for precise visual comprehension, representing a challenging and intelligent capability essential for practical applications like photography assistant agents. We aim to evaluate MLLMs on their ability to distinguish visual differences related to numerical camera settings, extending a methodology previously proposed for vision-language models (VLMs). Our preliminary results demonstrate the importance of visual reasoning in photography-related tasks. Moreover, these results show that no single MLLM consistently dominates across all evaluation tasks, demonstrating ongoing challenges and opportunities in developing MLLMs with better visual reasoning.

CameraBench: Benchmarking Visual Reasoning in MLLMs via Photography

TL;DR

This paper addresses visual reasoning in photography by introducing CameraBench, a benchmark that tests multimodal LLMs on inferring numerical camera settings from images. It builds CameraSettings25K, normalizes settings to 35mm full-frame, and defines binary-choice and five-choice tasks to probe distinctions across focal length, aperture, ISO, and exposure time. Results show that larger, reasoning-focused MLLMs perform better on certain tasks, but many models struggle with single-parameter distinctions, revealing gaps in visual reasoning for photography. The work underscores the need for physics-aware visual reasoning in practical photo assistant applications and lays groundwork for broader benchmarking across MLLMs.

Abstract

Large language models (LLMs) and multimodal large language models (MLLMs) have significantly advanced artificial intelligence. However, visual reasoning, reasoning involving both visual and textual inputs, remains underexplored. Recent advancements, including the reasoning models like OpenAI o1 and Gemini 2.0 Flash Thinking, which incorporate image inputs, have opened this capability. In this ongoing work, we focus specifically on photography-related tasks because a photo is a visual snapshot of the physical world where the underlying physics (i.e., illumination, blur extent, etc.) interplay with the camera parameters. Successfully reasoning from the visual information of a photo to identify these numerical camera settings requires the MLLMs to have a deeper understanding of the underlying physics for precise visual comprehension, representing a challenging and intelligent capability essential for practical applications like photography assistant agents. We aim to evaluate MLLMs on their ability to distinguish visual differences related to numerical camera settings, extending a methodology previously proposed for vision-language models (VLMs). Our preliminary results demonstrate the importance of visual reasoning in photography-related tasks. Moreover, these results show that no single MLLM consistently dominates across all evaluation tasks, demonstrating ongoing challenges and opportunities in developing MLLMs with better visual reasoning.

Paper Structure

This paper contains 4 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Data Curation and Benchmark
  3. Experimental Results
  4. Conclusions

Figures (2)

  • Figure 1: Radar chart of CameraBench. We evaluate MLLMs by performing binary-choice questions on a set of numerical camera settings associated with the given images. The questions are conducted under different conditions. All: The camera settings in the two options are all different. Focal Length: Only the focal length is different between the two options. Aperture: Only the aperture is different between the two options. ISO: Only the ISO speed rating is different between the two options. Exposure Time: Only the exposure time is different between the two options.
  • Figure 2: Example question in CameraBench. We provide two options of numerical camera settings, one is extracted from the exif of the raw image, another is sampled from other image in CameraSettings25K.