NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?

TL;DR

This work explores MLLMs' reasoning capabilities in object recognition, ranging from commonsense to beyond-commonsense scenarios, and introduces a novel benchmark, NEMO, which comprises 900 images of origiNal fruits and their corresponding attributE-MOdified ones; along with a set of 2,700 questions including open-, multiple-choice-, unsolvable types.

Abstract

Multimodal Large Language Models (MLLMs) have made notable advances in visual understanding, yet their abilities to recognize objects modified by specific attributes remain an open question. To address this, we explore MLLMs' reasoning capabilities in object recognition, ranging from commonsense to beyond-commonsense scenarios. We introduce a novel benchmark, NEMO, which comprises 900 images of origiNal fruits and their corresponding attributE-MOdified ones; along with a set of 2,700 questions including open-, multiple-choice-, unsolvable types. We assess 26 recent open-sourced and commercial models using our benchmark. The findings highlight pronounced performance gaps in recognizing objects in NEMO and reveal distinct answer preferences across different models. Although stronger vision encoders improve performance, MLLMs still lag behind standalone vision encoders. Interestingly, scaling up the model size does not consistently yield better outcomes, as deeper analysis reveals that larger LLMs can weaken vision encoders during fine-tuning. These insights shed light on critical limitations in current MLLMs and suggest potential pathways toward developing more versatile and resilient multimodal models.

Figures (16)

• Figure 1: Challenges in consistently identifying original and attribute-modified objects are illustrated in (a) and (b): while LLaVA-NeXT-Qwen-32B and GPT-4o correctly recognize the original "mango", both fail with a blue variant. Panels (c) and (d) show average accuracy scores of LLaVA-NeXT-Qwen-32B and GPT-4o across objects in our NEMO, comparing original and attribute-modified versions. (e) provides a comparative overview of average scores for representative MLLMs on original (upper) and attribute-modified (lower) objects.
• Figure 2: Samples from the NEMO benchmark, showcasing original and attribute-modified (A-Modif) objects with three types of question formats: open questions, multiple-choice questions, and unsolvable questions. We additionally circularly shift the choices 3 times in multiple-choice questions and unsolvable questions for extensive evaluation.
• Figure 3: Model accuracy scores across vision encoders for multiple-choice questions on original objects (upper) and attribute-modified objects (lower). MLLMs using the same vision encoder are shown in the same color, while vision encoders themselves are distinguished by diagonal line patterns. Stronger encoders generally improve MLLM performance, yet most MLLMs perform worse than their encoders.
• Figure 4: Average accuracy scores of various MLLMs (LLaVA-NeXT and InternVL2 series) for all question types across different model sizes on original and attribute-modified objects. The performance does not consistently improve as the model size increases.
• Figure 5: Top-5 accuracy of Original-to-Original and Attribute-modified-to-Original object matching, using image representations obtained from MLLMs' trained vision encoders. Fine-tuning with larger LLMs may weaken the vision encoders.