Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DIVE: Towards Descriptive and Diverse Visual Commonsense Generation

Jun-Hyung Park, Hyuntae Park, Youjin Kang, Eojin Jeon, SangKeun Lee

TL;DR

This paper tackles the gap in visual commonsense generation where prior work yields generic, limited inferences. It introduces DIVE, a framework with two components: generic inference filtering to balance the training data by removing overrepresented, vague inferences, and contrastive retrieval learning to teach models to identify image-specific details by aligning image-inference pairs within similar image sets. Empirical results on the Visual Commonsense Graph (VCG) show that DIVE significantly improves descriptiveness and diversity, achieving near-human performance on several metrics and surpassing strong baselines on unique and novel inferences; human evaluations corroborate these gains. The work advances practical visual reasoning by producing richer, more informative inferences, with implications for downstream vision-language tasks and multi-modal understanding, while noting data-filtering trade-offs and suggesting avenues for augmentation and cross-modal extension.

Abstract

Towards human-level visual understanding, visual commonsense generation has been introduced to generate commonsense inferences beyond images. However, current research on visual commonsense generation has overlooked an important human cognitive ability: generating descriptive and diverse inferences. In this work, we propose a novel visual commonsense generation framework, called DIVE, which aims to improve the descriptiveness and diversity of generated inferences. DIVE involves two methods, generic inference filtering and contrastive retrieval learning, which address the limitations of existing visual commonsense resources and training objectives. Experimental results verify that DIVE outperforms state-of-the-art models for visual commonsense generation in terms of both descriptiveness and diversity, while showing a superior quality in generating unique and novel inferences. Notably, DIVE achieves human-level descriptiveness and diversity on Visual Commonsense Graphs. Furthermore, human evaluations confirm that DIVE aligns closely with human judgments on descriptiveness and diversity\footnote{Our code and dataset are available at https://github.com/Park-ing-lot/DIVE.

DIVE: Towards Descriptive and Diverse Visual Commonsense Generation

TL;DR

This paper tackles the gap in visual commonsense generation where prior work yields generic, limited inferences. It introduces DIVE, a framework with two components: generic inference filtering to balance the training data by removing overrepresented, vague inferences, and contrastive retrieval learning to teach models to identify image-specific details by aligning image-inference pairs within similar image sets. Empirical results on the Visual Commonsense Graph (VCG) show that DIVE significantly improves descriptiveness and diversity, achieving near-human performance on several metrics and surpassing strong baselines on unique and novel inferences; human evaluations corroborate these gains. The work advances practical visual reasoning by producing richer, more informative inferences, with implications for downstream vision-language tasks and multi-modal understanding, while noting data-filtering trade-offs and suggesting avenues for augmentation and cross-modal extension.

Abstract

Towards human-level visual understanding, visual commonsense generation has been introduced to generate commonsense inferences beyond images. However, current research on visual commonsense generation has overlooked an important human cognitive ability: generating descriptive and diverse inferences. In this work, we propose a novel visual commonsense generation framework, called DIVE, which aims to improve the descriptiveness and diversity of generated inferences. DIVE involves two methods, generic inference filtering and contrastive retrieval learning, which address the limitations of existing visual commonsense resources and training objectives. Experimental results verify that DIVE outperforms state-of-the-art models for visual commonsense generation in terms of both descriptiveness and diversity, while showing a superior quality in generating unique and novel inferences. Notably, DIVE achieves human-level descriptiveness and diversity on Visual Commonsense Graphs. Furthermore, human evaluations confirm that DIVE aligns closely with human judgments on descriptiveness and diversity\footnote{Our code and dataset are available at https://github.com/Park-ing-lot/DIVE.
Paper Structure (26 sections, 6 equations, 10 figures, 17 tables)

This paper contains 26 sections, 6 equations, 10 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual commonsense reasoning.
  4. Descriptive and diverse text generation.
  5. Methodology
  6. Generic Inference Filtering
  7. Contrastive Retrieval Learning
  8. Experiments
  9. Experimental Setup
  10. Dataset.
  11. Baselines.
  12. Implementation details.
  13. Metrics.
  14. Main Results
  15. Human Evaluation Results
  16. ...and 11 more sections

Figures (10)

  • Figure 1: Comparison of commonsense inferences from models and humans. Blue words represent key details.
  • Figure 2: Illustration of DIVE. (a) Generic Inference Filtering: Filtering out inferences with high frequency and low semantic concentration of related images. (b) Contrastive Retrieval Learning: Learning to maximize the agreement between a pair of an image and its unique corresponding inference. Events and places are omitted for clarity.
  • Figure 3: Illustration of measuring the semantic concentration and frequency of inferences.
  • Figure 4: Distribution of generated inferences in relation to word-level entropy.
  • Figure 5: Comparison of generation examples from DIVE, KM-BART kmbart, VisualCOMET visualcomet, and human annotations in VCG validation set. We mark red if the inference is implausible and blue if the inference is both unique and novel.
  • ...and 5 more figures