Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal LLM Augmented Reasoning for Interpretable Visual Perception Analysis

Shravan Chaudhari, Trilokya Akula, Yoon Kim, Tom Blake

TL;DR

This work addresses the need for principled evaluation of multimodal LLMs in interpretable visual perception tasks within HCI. It proposes an annotation-free analytical framework that embeds Gestalt-based explainable parameters and uses pairwise comparisons to quantify perceptual complexity via a capable MLLM (Claude Sonnet 3.0) across SAVOIAS and IC9600 datasets, evaluated with PLCC and SROCC. The key finding is that visual clutter and the law of simplicity most strongly align with human judgments of visual complexity, while other gestalt principles show variable correlations, indicating biases in human annotations. The framework offers a scalable, bias-resistant approach to benchmarking MLLMs for cognitive assistance in HCI, with broad implications for UI/UX design, accessibility, and content presentation, and sets the stage for cross-model benchmarking and expanded perceptual-principle analyses.

Abstract

In this paper, we advance the study of AI-augmented reasoning in the context of Human-Computer Interaction (HCI), psychology and cognitive science, focusing on the critical task of visual perception. Specifically, we investigate the applicability of Multimodal Large Language Models (MLLMs) in this domain. To this end, we leverage established principles and explanations from psychology and cognitive science related to complexity in human visual perception. We use them as guiding principles for the MLLMs to compare and interprete visual content. Our study aims to benchmark MLLMs across various explainability principles relevant to visual perception. Unlike recent approaches that primarily employ advanced deep learning models to predict complexity metrics from visual content, our work does not seek to develop a mere new predictive model. Instead, we propose a novel annotation-free analytical framework to assess utility of MLLMs as cognitive assistants for HCI tasks, using visual perception as a case study. The primary goal is to pave the way for principled study in quantifying and evaluating the interpretability of MLLMs for applications in improving human reasoning capability and uncovering biases in existing perception datasets annotated by humans.

Multimodal LLM Augmented Reasoning for Interpretable Visual Perception Analysis

TL;DR

This work addresses the need for principled evaluation of multimodal LLMs in interpretable visual perception tasks within HCI. It proposes an annotation-free analytical framework that embeds Gestalt-based explainable parameters and uses pairwise comparisons to quantify perceptual complexity via a capable MLLM (Claude Sonnet 3.0) across SAVOIAS and IC9600 datasets, evaluated with PLCC and SROCC. The key finding is that visual clutter and the law of simplicity most strongly align with human judgments of visual complexity, while other gestalt principles show variable correlations, indicating biases in human annotations. The framework offers a scalable, bias-resistant approach to benchmarking MLLMs for cognitive assistance in HCI, with broad implications for UI/UX design, accessibility, and content presentation, and sets the stage for cross-model benchmarking and expanded perceptual-principle analyses.

Abstract

In this paper, we advance the study of AI-augmented reasoning in the context of Human-Computer Interaction (HCI), psychology and cognitive science, focusing on the critical task of visual perception. Specifically, we investigate the applicability of Multimodal Large Language Models (MLLMs) in this domain. To this end, we leverage established principles and explanations from psychology and cognitive science related to complexity in human visual perception. We use them as guiding principles for the MLLMs to compare and interprete visual content. Our study aims to benchmark MLLMs across various explainability principles relevant to visual perception. Unlike recent approaches that primarily employ advanced deep learning models to predict complexity metrics from visual content, our work does not seek to develop a mere new predictive model. Instead, we propose a novel annotation-free analytical framework to assess utility of MLLMs as cognitive assistants for HCI tasks, using visual perception as a case study. The primary goal is to pave the way for principled study in quantifying and evaluating the interpretability of MLLMs for applications in improving human reasoning capability and uncovering biases in existing perception datasets annotated by humans.

Paper Structure

This paper contains 11 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Method
  5. Prompt Design
  6. Quantifying reasoning-based visual perception analysis
  7. Correlation with complexity in human perception
  8. MLLM Models
  9. Results & Discussion
  10. Applications & Future Work
  11. Appendix

Figures (5)

  • Figure 1: Images ordered based on visual complexity across different categories.
  • Figure 2: Visual perception analysis framework using multimodal LLMs
  • Figure 3: Various important components of the prompt to compare advertisement images based on visual clutter.
  • Figure 4: Sample MLLM (Claude Sonnet 3.0) output comparing two interior design images.
  • Figure 5: Results summary for SAVIOAS and IC9600 dataset. Detailed results can be found in tables \ref{['table:savoias_results']} and \ref{['table:ic9600_results']} in the appendix.