Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction

Henry Peng Zou, Vinay Samuel, Yue Zhou, Weizhi Zhang, Liancheng Fang, Zihe Song, Philip S. Yu, Cornelia Caragea

TL;DR

ImplicitAVE addresses the gap in implicit attribute value extraction by providing the first publicly available multimodal dataset that pairs product text with images across five domains. The authors construct ImplicitAVE through data collection from MAVE, rigorous curation, implicit-value expansion, and two-stage human validation, yielding 68,604 training and 1,610 test instances with 25 attributes and 158 values. They benchmark six recent multimodal LLMs (11 variants) under zero-shot and fine-tuned settings, revealing that implicit AVE remains challenging and that GPT-4V currently offers the strongest performance, with open-source models lagging behind. The dataset and benchmarks enable systematic study of multimodal reasoning for implicit attribute inference and pave the way for future work on multi-valued attributes and negative none values in e-commerce scenarios.

Abstract

Existing datasets for attribute value extraction (AVE) predominantly focus on explicit attribute values while neglecting the implicit ones, lack product images, are often not publicly available, and lack an in-depth human inspection across diverse domains. To address these limitations, we present ImplicitAVE, the first, publicly available multimodal dataset for implicit attribute value extraction. ImplicitAVE, sourced from the MAVE dataset, is carefully curated and expanded to include implicit AVE and multimodality, resulting in a refined dataset of 68k training and 1.6k testing data across five domains. We also explore the application of multimodal large language models (MLLMs) to implicit AVE, establishing a comprehensive benchmark for MLLMs on the ImplicitAVE dataset. Six recent MLLMs with eleven variants are evaluated across diverse settings, revealing that implicit value extraction remains a challenging task for MLLMs. The contributions of this work include the development and release of ImplicitAVE, and the exploration and benchmarking of various MLLMs for implicit AVE, providing valuable insights and potential future research directions. Dataset and code are available at https://github.com/HenryPengZou/ImplicitAVE

ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction

TL;DR

ImplicitAVE addresses the gap in implicit attribute value extraction by providing the first publicly available multimodal dataset that pairs product text with images across five domains. The authors construct ImplicitAVE through data collection from MAVE, rigorous curation, implicit-value expansion, and two-stage human validation, yielding 68,604 training and 1,610 test instances with 25 attributes and 158 values. They benchmark six recent multimodal LLMs (11 variants) under zero-shot and fine-tuned settings, revealing that implicit AVE remains challenging and that GPT-4V currently offers the strongest performance, with open-source models lagging behind. The dataset and benchmarks enable systematic study of multimodal reasoning for implicit attribute inference and pave the way for future work on multi-valued attributes and negative none values in e-commerce scenarios.

Abstract

Existing datasets for attribute value extraction (AVE) predominantly focus on explicit attribute values while neglecting the implicit ones, lack product images, are often not publicly available, and lack an in-depth human inspection across diverse domains. To address these limitations, we present ImplicitAVE, the first, publicly available multimodal dataset for implicit attribute value extraction. ImplicitAVE, sourced from the MAVE dataset, is carefully curated and expanded to include implicit AVE and multimodality, resulting in a refined dataset of 68k training and 1.6k testing data across five domains. We also explore the application of multimodal large language models (MLLMs) to implicit AVE, establishing a comprehensive benchmark for MLLMs on the ImplicitAVE dataset. Six recent MLLMs with eleven variants are evaluated across diverse settings, revealing that implicit value extraction remains a challenging task for MLLMs. The contributions of this work include the development and release of ImplicitAVE, and the exploration and benchmarking of various MLLMs for implicit AVE, providing valuable insights and potential future research directions. Dataset and code are available at https://github.com/HenryPengZou/ImplicitAVE
Paper Structure (39 sections, 10 figures, 7 tables)

This paper contains 39 sections, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Dataset Construction
  3. Initial Data Collection
  4. Data Curation for Implicit AVE
  5. Data Expansion
  6. Human Inspection & Re-Annotation
  7. Dataset Statistics
  8. Experiment & Benchmark
  9. Experimental Setting
  10. Evaluation Setup
  11. Models for Zero-Shot
  12. Models for Finetuning
  13. Experimental Results
  14. Domain-Level Results
  15. (1) Attributes within the Clothing domain demand a more nuanced understanding of local details in product images.
  16. ...and 24 more sections

Figures (10)

  • Figure 1: An example of implicit attribute value. The attribute value "Rain Boot" is not mentioned explicitly in the product text, but can be inferred from text context, product image, or prior knowledge.
  • Figure 2: Steps for constructing our ImplicitAVE dataset. A detailed explanation is provided in Section \ref{['sec:dataset_construction']}.
  • Figure 3: Data distribution of domains, attributes, and attribute values for training and evaluation sets. (A full-size version is attached to our appendix - Figure \ref{['fig:appendix_large_visual_data_distribution']})
  • Figure 4: Performance comparison in few-shot settings of different domains.
  • Figure 5: Performance comparison of DEFLATE, LAVIN, and GPT-4V on different modalities.
  • ...and 5 more figures