Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVA

TL;DR

The paper investigates toxicity in image-text pretraining datasets for vision-language models, focusing on LLaVA. It presents a multimodal toxicity-detection pipeline using LlavaGuard for images and Toxic-BERT for text, augmented by a Cohere prompt tuner and Command R+ to refine results. 7,531 toxic image-text pairs are removed to produce a toxicity-mitigated LLaVA pretraining dataset, which is released as open-source. The work also provides guidelines for toxicity evaluation and suggests integrating safety frameworks such as SPA-VL, SafeCLIP, MM-SafetyBench, VHELM, and ELITE to enable safer multimodal AI.

Abstract

Pretraining datasets are foundational to the development of multimodal models, yet they often have inherent biases and toxic content from the web-scale corpora they are sourced from. In this paper, we investigate the prevalence of toxicity in LLaVA image-text pretraining dataset, examining how harmful content manifests in different modalities. We present a comprehensive analysis of common toxicity categories and propose targeted mitigation strategies, resulting in the creation of a refined toxicity-mitigated dataset. This dataset removes 7,531 of toxic image-text pairs in the LLaVA pre-training dataset. We offer guidelines for implementing robust toxicity detection pipelines. Our findings underscore the need to actively identify and filter toxic content - such as hate speech, explicit imagery, and targeted harassment - to build more responsible and equitable multimodal systems. The toxicity-mitigated dataset is open source and is available for further research.

Figures (3)

• Figure 1: Image Toxicity Analysis on LLaVA Pre-train Dataset using LlavaGuard
• Figure 2: Image Caption Toxicity Analysis on LLaVA Pre-train Dataset using Toxic-BERT
• Figure 3: Dataset Toxicity Filtering Method