Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models

TL;DR

This paper introduces the Multi-Dimensional Insights (MDI) benchmark to evaluate large multimodal models on real-world personalization needs. It assembles 514 new images and 1,298 questions across six life scenarios, with two complexity levels and an age-stratified design (young, middle-aged, old) to probe cross-group capabilities. Evaluating 14 models, including GPT-4o, reveals that while GPT-4o achieves the best overall performance, significant gaps remain across scenarios and age groups, and open-source models lag behind closed-source ones. The benchmark, along with its data and evaluation code, aims to drive the development of LMMs that can more robustly align with diverse human needs in everyday contexts.

Abstract

The rapidly developing field of large multimodal models (LMMs) has led to the emergence of diverse models with remarkable capabilities. However, existing benchmarks fail to comprehensively, objectively and accurately evaluate whether LMMs align with the diverse needs of humans in real-world scenarios. To bridge this gap, we propose the Multi-Dimensional Insights (MDI) benchmark, which includes over 500 images covering six common scenarios of human life. Notably, the MDI-Benchmark offers two significant advantages over existing evaluations: (1) Each image is accompanied by two types of questions: simple questions to assess the model's understanding of the image, and complex questions to evaluate the model's ability to analyze and reason beyond basic content. (2) Recognizing that people of different age groups have varying needs and perspectives when faced with the same scenario, our benchmark stratifies questions into three age categories: young people, middle-aged people, and older people. This design allows for a detailed assessment of LMMs' capabilities in meeting the preferences and needs of different age groups. With MDI-Benchmark, the strong model like GPT-4o achieve 79% accuracy on age-related tasks, indicating that existing LMMs still have considerable room for improvement in addressing real-world applications. Looking ahead, we anticipate that the MDI-Benchmark will open new pathways for aligning real-world personalization in LMMs. The MDI-Benchmark data and evaluation code are available at https://mdi-benchmark.github.io/

Figures (33)

• Figure 1: The MDI-Benchmark includes real needs of different age groups in six major real-world scenarios.
• Figure 2: The overview of the MDI Benchmark's six real-world multimodal scenarios, each comprising three sub-domains.
• Figure 3: The average performance of different LMMs on different difficulty levels of the MDI-Benchmark.
• Figure 4: Performance of the model at different difficulty levels and the overall performance results of the model under the score metric.
• Figure 5: The average accuracy and variance of LLMs across six domains at Level 1 and Level 2