InfoAffect: A Dataset for Affective Analysis of Infographics

TL;DR

InfoAffect addresses the lack of affective analysis resources for infographics by introducing a 3.5k-sample dual-modal dataset that pairs real-world infographics with accompanying text and constrains affect labels via an Affect Table. The pipeline collects data from six domains, applies quality controls, and uses five multimodal large language models to extract affects, fused with Reciprocal Rank Fusion. Two user studies validate usability and accuracy, reporting a Composite Affect Consistency Index of 0.986, indicating strong alignment with human judgments. The resource supports robust affective modeling of infographics and can facilitate affect-aware design and downstream fine-tuning of multimodal models.

Abstract

Infographics are widely used to convey complex information, yet their affective dimensions remain underexplored due to the scarcity of data resources. We introduce a 3.5k-sample affect-annotated InfoAffect dataset, which combines textual content with real-world infographics. We first collect the raw data from six domains and aligned them via preprocessing, the accompanied-text-priority method, and three strategies to guarantee the quality and compliance. After that we construct an affect table and use it to constrain annotation. Five state-of-the-art multimodal large language models (MLLMs) then analyze both modalities, and their outputs are fused with Reciprocal Rank Fusion (RRF) algorithm to yield robust affects and confidences. We conducted a user study with two experiments to validate usability and assess InfoAffect dataset using the Composite Affect Consistency Index (CACI), achieving an overall score of 0.986, which indicates high accuracy.

Figures (5)

• Figure 1: Overview of the InfoAffect dataset construction process. For raw data, we algin it through three methods, then extract affects with five state-of-the-art MLLMs, each model returns JSON with affective words, and their confidence. Finally fuse results using RRF algorithm.
• Figure 2: Overview of the Affect Table construction process.
• Figure 3: The innermost layer represents the three primary affective polarities classified by first-level affective classification, while the outer rings display the finer second-level affective classification that form the Affect Table. Each sector's area corresponds to the relative frequency of affects, revealing a balanced and comprehensive affective coverage across different polarities.
• Figure 4: Detailed illustration of the Reciprocal Rank Fusion (RRF) algorithm applied in affective result fusion.
• Figure 5: Visual comparison of CACI results between InfoAffect dataset and crowdsourced ratings. Panels (a–i) were the representative examples cited in this section (labels on each panel); they illustrate typical cases rather than the full dataset. Each row corresponds to one affective category, indicated by colored legend markers: $\blacksquare$Negative, $\blacksquare$Neutral, and $\blacksquare$Positive.