LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences

TL;DR

This work introduces LOTUS, a unified leaderboard for evaluating detailed image captions produced by LVLMs, addressing the lack of standardized criteria, bias awareness, and user-preference considerations. LOTUS combines four core quality criteria (alignment, descriptiveness, language complexity, side effects) with bias and language-discrepancy analyses, using multiple metrics and a normalized average to enable holistic model comparisons. Experiments across five LVLMs reveal that no model excels across all criteria, with distinct trade-offs between descriptiveness and bias/hallucination risks, and demonstrate that user preferences significantly influence optimal model selection. The framework also emphasizes ethical considerations and provides guidelines for fairness, including limitations of binary demographic categorizations and the need for richer attribute representations in future work.

Abstract

Large Vision-Language Models (LVLMs) have transformed image captioning, shifting from concise captions to detailed descriptions. We introduce LOTUS, a leaderboard for evaluating detailed captions, addressing three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. LOTUS comprehensively evaluates various aspects, including caption quality (e.g., alignment, descriptiveness), risks (\eg, hallucination), and societal biases (e.g., gender bias) while enabling preference-oriented evaluations by tailoring criteria to diverse user preferences. Our analysis of recent LVLMs reveals no single model excels across all criteria, while correlations emerge between caption detail and bias risks. Preference-oriented evaluations demonstrate that optimal model selection depends on user priorities.

Figures (11)

• Figure 1: Overview of the LOTUS leaderboard. LOTUS enables (a) unified evaluation of various aspects of detailed captions, including societal bias, and (b) preference-oriented assessment tailored to different user preferences.
• Figure 2: Correlation matrix of evaluation criteria.
• Figure 3: Gender and skin tone representations in generated captions. Rec$_\text{F/M}$ denotes recall of gender terms for woman/man images. Rec$_\text{D/L}$ represents recall of racial terms for darker/lighter skin. $|\Delta|$ is recall disparities.
• Figure 4: Preference-oriented scores for detail-oriented user (left), risk-conscious user (middle), and accuracy-focused user (right). The best models for each user type are highlighted in darker colors.
• Figure 5: Simulated user prompts for each user type.