EIVEN: Efficient Implicit Attribute Value Extraction using Multimodal LLM

TL;DR

The paper tackles implicit attribute value extraction from multimodal e-commerce data, where values may be inferred from images, text context, or prior knowledge. It introduces EIVEN, a data- and parameter-efficient framework that leverages a frozen vision encoder and a pretrained LLM (LLaMA-7B) with lightweight adapters, enhanced by a Learning-by-Comparison mechanism to reduce confusion among similar values. EIVEN also constructs open-source datasets (Clothing, Footwear, General) for multimodal implicit AVE, derived from MAVE and Amazon Reviews, and demonstrates significant improvements over baselines (DEFLATE, CMA-CLIP, M-JAVE) with far fewer labeled examples. The approach yields practical benefits for e-commerce attribute extraction by reducing labeling needs and computation, while advancing open benchmarks for implicit AVE research.

Abstract

In e-commerce, accurately extracting product attribute values from multimodal data is crucial for improving user experience and operational efficiency of retailers. However, previous approaches to multimodal attribute value extraction often struggle with implicit attribute values embedded in images or text, rely heavily on extensive labeled data, and can easily confuse similar attribute values. To address these issues, we introduce EIVEN, a data- and parameter-efficient generative framework that pioneers the use of multimodal LLM for implicit attribute value extraction. EIVEN leverages the rich inherent knowledge of a pre-trained LLM and vision encoder to reduce reliance on labeled data. We also introduce a novel Learning-by-Comparison technique to reduce model confusion by enforcing attribute value comparison and difference identification. Additionally, we construct initial open-source datasets for multimodal implicit attribute value extraction. Our extensive experiments reveal that EIVEN significantly outperforms existing methods in extracting implicit attribute values while requiring less labeled data.

Figures (11)

• Figure 1: Examples of implicit attribute values. The attribute value cannot be explicitly extracted as a part of product texts, but can inferred from the product image, text context or prior knowledge.
• Figure 2: Overview of our efficient multimodal LLM. We extract multi-granularity visual features from a frozen pre-trained vision encoder and use a learnable visual projection network to align their dimensions with text token embeddings. The obtained visual tokens and tokenized question and text context are fed to the LLM (LLaMA-7B) to generate the answer. We insert lightweight adapters into every layer of the LLM for parameter-efficient fine-tuning.
• Figure 3: Illustration of Learning-by-Comparison strategies. Our model is fed with pairs of product instances that share the same attribute but potentially different attribute values and asked to compare the values.
• Figure 4: Data efficiency demonstration with varying numbers of labeled data. EIVEN can achieve better performance than DEFLATE with less labeled data, highlighting its data efficiency.
• Figure 5: Qualitative examples and comparisons between EIVEN and DEFLATE.