Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models

Zhongzhan Huang, Shanshan Zhong, Pan Zhou, Shanghua Gao, Marinka Zitnik, Liang Lin

TL;DR

The results show that while most LLMs exhibit constrained creativity, the performance gap between LLMs and humans is not insurmountable, and suggest that LoTbench better aligns with human cognitive theories, highlighting cognition as a critical foundation in the early stages of creativity and enabling the bridging of diverse concepts.

Abstract

Recently, numerous benchmarks have been developed to evaluate the logical reasoning abilities of large language models (LLMs). However, assessing the equally important creative capabilities of LLMs is challenging due to the subjective, diverse, and data-scarce nature of creativity, especially in multimodal scenarios. In this paper, we consider the comprehensive pipeline for evaluating the creativity of multimodal LLMs, with a focus on suitable evaluation platforms and methodologies. First, we find the Oogiri game, a creativity-driven task requiring humor, associative thinking, and the ability to produce unexpected responses to text, images, or both. This game aligns well with the input-output structure of modern multimodal LLMs and benefits from a rich repository of high-quality, human-annotated creative responses, making it an ideal platform for studying LLM creativity. Next, beyond using the Oogiri game for standard evaluations like ranking and selection, we propose LoTbench, an interactive, causality-aware evaluation framework, to further address some intrinsic risks in standard evaluations, such as information leakage and limited interpretability. The proposed LoTbench not only quantifies LLM creativity more effectively but also visualizes the underlying creative thought processes. Our results show that while most LLMs exhibit constrained creativity, the performance gap between LLMs and humans is not insurmountable. Furthermore, we observe a strong correlation between results from the multimodal cognition benchmark MMMU and LoTbench, but only a weak connection with traditional creativity metrics. This suggests that LoTbench better aligns with human cognitive theories, highlighting cognition as a critical foundation in the early stages of creativity and enabling the bridging of diverse concepts. https://lotbench.github.io

A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models

TL;DR

The results show that while most LLMs exhibit constrained creativity, the performance gap between LLMs and humans is not insurmountable, and suggest that LoTbench better aligns with human cognitive theories, highlighting cognition as a critical foundation in the early stages of creativity and enabling the bridging of diverse concepts.

Abstract

Recently, numerous benchmarks have been developed to evaluate the logical reasoning abilities of large language models (LLMs). However, assessing the equally important creative capabilities of LLMs is challenging due to the subjective, diverse, and data-scarce nature of creativity, especially in multimodal scenarios. In this paper, we consider the comprehensive pipeline for evaluating the creativity of multimodal LLMs, with a focus on suitable evaluation platforms and methodologies. First, we find the Oogiri game, a creativity-driven task requiring humor, associative thinking, and the ability to produce unexpected responses to text, images, or both. This game aligns well with the input-output structure of modern multimodal LLMs and benefits from a rich repository of high-quality, human-annotated creative responses, making it an ideal platform for studying LLM creativity. Next, beyond using the Oogiri game for standard evaluations like ranking and selection, we propose LoTbench, an interactive, causality-aware evaluation framework, to further address some intrinsic risks in standard evaluations, such as information leakage and limited interpretability. The proposed LoTbench not only quantifies LLM creativity more effectively but also visualizes the underlying creative thought processes. Our results show that while most LLMs exhibit constrained creativity, the performance gap between LLMs and humans is not insurmountable. Furthermore, we observe a strong correlation between results from the multimodal cognition benchmark MMMU and LoTbench, but only a weak connection with traditional creativity metrics. This suggests that LoTbench better aligns with human cognitive theories, highlighting cognition as a critical foundation in the early stages of creativity and enabling the bridging of diverse concepts. https://lotbench.github.io
Paper Structure (30 sections, 8 equations, 16 figures, 4 tables, 1 algorithm)

This paper contains 30 sections, 8 equations, 16 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Evaluation Platform: Oogiri game
  4. Oogiri-GO Dataset
  5. Standard Evaluation with Oogiri game
  6. LoTbench with Oogiri game
  7. The formulation of LoTbench
  8. Tuning LLM for Data Synthesis by CLoTv2
  9. Associable Instruction Tuning
  10. Explorative Self-Refinement
  11. The Data Construction in LoTbench
  12. The Details of $I_t$ and $R_t$
  13. To Measure DAESO with Causal Evaluator $\mathcal{E}_1$?
  14. Criteria
  15. Modeling
  16. ...and 15 more sections

Figures (16)

  • Figure 1: Leap-of-Thought (LoT) for creativity. (a) Comparison of CoT and LoT. "$\bigcirc$" denotes the thought and "→" represents the connection between two thoughts. LoT is one of most important ability in creativity talmor2020leapcallaway2013cognitive. (b) Examples of the three types of LoT-based Oogiri games. Players are required to make surprising and creative humorous responses (blue box) to the given multimodal information e.g., images, text, or both.
  • Figure 2: The motivation of different paradigms to measure creativity. (Left) Standard Evaluation: Assess LLMs by designing selection and ranking tasks. Higher accuracy indicates greater creativity. (Right) LoTbench: LLMs generate multi-round responses, evaluated by a causal evaluator to determine whether they approach high-quality human-level creative responses (HHCRs). If not, the model enters a rethinking phase for the next round. Creativity is inversely proportional to the number of response rounds (# Round).
  • Figure 3: The overview of proposed interactive creativity evaluation LoTbench for LLM. The main task in LoTbench is masked language modeling (MLM) task. DAESO denotes "different approach but equally satisfactory outcome", where $\mathcal{E}_1$ is the causal evaluator for check whether $R_t$ and $R$ are DAESO.
  • Figure 4: The main task in LoTbench is masked language modeling (MLM). The LLMs are required to fill in the < MASK> in the sentence to make it a creative response relative to the provided image.
  • Figure 5: The details of LoT-oriented instructions templates. We take "Image to Text" as an example, see the Appendix of the conference version zhong2024let for the details of other categories' instructions. (a) and (b) are the instruction templates with/without conditions for associable generation. (c) and (d) are the two instructions about the selection and ranking of associable discrimination. All templates follow the formats in Fig. \ref{['fig:template']}.
  • ...and 11 more figures