Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Cross-Modal Error Detection with Tables and Images

Olga Ovcharenko, Sebastian Schelter

TL;DR

The paper tackles cross-modal error detection in tabular data paired with images, a practical need for multi-modal data pipelines in domains like e-Commerce. It formalizes the problem, builds a four-dataset benchmark, and evaluates five baselines spanning tabular detectors, label-error methods, and LLM/vision-language approaches. Findings show that label-error methods combined with AutoML (e.g., AutoGluon+Cleanlab, AutoGluon+DataScope) often achieve the highest F1, especially when leveraging both table and image data, but performance deteriorates on heavy-tailed real-world data and LLM/VLM methods exhibit inconsistent results and high costs. The work highlights a gap in current approaches and motivates developing dedicated multi-modal validation models and richer benchmarks to enable scalable, modality-aware data quality assurance in practice.

Abstract

Ensuring data quality at scale remains a persistent challenge for large organizations. Despite recent advances, maintaining accurate and consistent data is still complex, especially when dealing with multiple data modalities. Traditional error detection and correction methods tend to focus on a single modality, typically a table, and often miss cross-modal errors that are common in domains like e-Commerce and healthcare, where image, tabular, and text data co-exist. To address this gap, we take an initial step towards cross-modal error detection in tabular data, by benchmarking several methods. Our evaluation spans four datasets and five baseline approaches. Among them, Cleanlab, a label error detection framework, and DataScope, a data valuation method, perform the best when paired with a strong AutoML framework, achieving the highest F1 scores. Our findings indicate that current methods remain limited, particularly when applied to heavy-tailed real-world data, motivating further research in this area.

Towards Cross-Modal Error Detection with Tables and Images

TL;DR

The paper tackles cross-modal error detection in tabular data paired with images, a practical need for multi-modal data pipelines in domains like e-Commerce. It formalizes the problem, builds a four-dataset benchmark, and evaluates five baselines spanning tabular detectors, label-error methods, and LLM/vision-language approaches. Findings show that label-error methods combined with AutoML (e.g., AutoGluon+Cleanlab, AutoGluon+DataScope) often achieve the highest F1, especially when leveraging both table and image data, but performance deteriorates on heavy-tailed real-world data and LLM/VLM methods exhibit inconsistent results and high costs. The work highlights a gap in current approaches and motivates developing dedicated multi-modal validation models and richer benchmarks to enable scalable, modality-aware data quality assurance in practice.

Abstract

Ensuring data quality at scale remains a persistent challenge for large organizations. Despite recent advances, maintaining accurate and consistent data is still complex, especially when dealing with multiple data modalities. Traditional error detection and correction methods tend to focus on a single modality, typically a table, and often miss cross-modal errors that are common in domains like e-Commerce and healthcare, where image, tabular, and text data co-exist. To address this gap, we take an initial step towards cross-modal error detection in tabular data, by benchmarking several methods. Our evaluation spans four datasets and five baseline approaches. Among them, Cleanlab, a label error detection framework, and DataScope, a data valuation method, perform the best when paired with a strong AutoML framework, achieving the highest F1 scores. Our findings indicate that current methods remain limited, particularly when applied to heavy-tailed real-world data, motivating further research in this area.
Paper Structure (14 sections, 2 figures, 15 tables)

This paper contains 14 sections, 2 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Cross-Modal Error Detection in Tables
  4. Related Work
  5. Preliminary Experimental Results
  6. Data and Error Generation
  7. Baseline Error Detection Performance
  8. Results and Discussion
  9. Automated Repair
  10. Conclusions
  11. Appendix
  12. Tabular Data Generation
  13. LLM Prompting for Error Detection
  14. Supplementary Tables

Figures (2)

  • Figure 1: A real-world cross-modal error from an e-Commerce catalog where a toy with a choking hazard is advertised to children of a wrong age group. The tabular product data erroneously states that the LEGO toy is suitable for four month old babies, even though the product image indicates that it is meant for teenagers with 16+ years of age (accessed May 18, 2025).
  • Figure 2: Additional real-world examples of cross-modal errors in Amazon's tabular product metadata, which are obvious from the corresponding product images: A television is misleadingly labeled as 4K ready when the product image shows that its resolution is too low for 4K; A pink toilet seat is listed as having color "white"; A movie is marked as not rated, even though its cover clearly indicates that it is meant for adult audiences only; (product pages accessed on May 10, 2025).