Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RU-AI: A Large Multimodal Dataset for Machine-Generated Content Detection

Liting Huang, Zhihao Zhang, Yiran Zhang, Xiyue Zhou, Shoujin Wang

TL;DR

This paper introduces RU-AI, the first large-scale dataset that aligns real and machine-generated content across text, image, and voice, constructed by fusing three public caption datasets with five generation models per modality and a noise-augmented variant for robustness. The authors detail data construction strategies (caption selection, diffusion-based image generation, TTS voice synthesis), cross-modality alignment, and comprehensive data characteristics, including text and audio quality metrics and image labeling. They then propose a unified baseline using pre-trained multimodal encoders (ImageBind and LanguageBind) followed by individual MLP classifiers to detect origin across modalities, and demonstrate that LanguageBind generally yields stronger performance, with multi-modality input offering benefits while noise degrades accuracy by a small margin. Overall, RU-AI exposes the current limits of SOTA triple-modal detectors, especially under noise, and provides a public resource to advance robust detection methods for machine-generated content in the era of generative AI.

Abstract

The recent generative AI models' capability of creating realistic and human-like content is significantly transforming the ways in which people communicate, create and work. The machine-generated content is a double-edged sword. On one hand, it can benefit the society when used appropriately. On the other hand, it may mislead people, posing threats to the society, especially when mixed together with natural content created by humans. Hence, there is an urgent need to develop effective methods to detect machine-generated content. However, the lack of aligned multimodal datasets inhibited the development of such methods, particularly in triple-modality settings (e.g., text, image, and voice). In this paper, we introduce RU-AI, a new large-scale multimodal dataset for robust and effective detection of machine-generated content in text, image and voice. Our dataset is constructed on the basis of three large publicly available datasets: Flickr8K, COCO and Places205, by adding their corresponding AI duplicates, resulting in a total of 1,475,370 instances. In addition, we created an additional noise variant of the dataset for testing the robustness of detection models. We conducted extensive experiments with the current SOTA detection methods on our dataset. The results reveal that existing models still struggle to achieve accurate and robust detection on our dataset. We hope that this new data set can promote research in the field of machine-generated content detection, fostering the responsible use of generative AI. The source code and datasets are available at https://github.com/ZhihaoZhang97/RU-AI.

RU-AI: A Large Multimodal Dataset for Machine-Generated Content Detection

TL;DR

This paper introduces RU-AI, the first large-scale dataset that aligns real and machine-generated content across text, image, and voice, constructed by fusing three public caption datasets with five generation models per modality and a noise-augmented variant for robustness. The authors detail data construction strategies (caption selection, diffusion-based image generation, TTS voice synthesis), cross-modality alignment, and comprehensive data characteristics, including text and audio quality metrics and image labeling. They then propose a unified baseline using pre-trained multimodal encoders (ImageBind and LanguageBind) followed by individual MLP classifiers to detect origin across modalities, and demonstrate that LanguageBind generally yields stronger performance, with multi-modality input offering benefits while noise degrades accuracy by a small margin. Overall, RU-AI exposes the current limits of SOTA triple-modal detectors, especially under noise, and provides a public resource to advance robust detection methods for machine-generated content in the era of generative AI.

Abstract

The recent generative AI models' capability of creating realistic and human-like content is significantly transforming the ways in which people communicate, create and work. The machine-generated content is a double-edged sword. On one hand, it can benefit the society when used appropriately. On the other hand, it may mislead people, posing threats to the society, especially when mixed together with natural content created by humans. Hence, there is an urgent need to develop effective methods to detect machine-generated content. However, the lack of aligned multimodal datasets inhibited the development of such methods, particularly in triple-modality settings (e.g., text, image, and voice). In this paper, we introduce RU-AI, a new large-scale multimodal dataset for robust and effective detection of machine-generated content in text, image and voice. Our dataset is constructed on the basis of three large publicly available datasets: Flickr8K, COCO and Places205, by adding their corresponding AI duplicates, resulting in a total of 1,475,370 instances. In addition, we created an additional noise variant of the dataset for testing the robustness of detection models. We conducted extensive experiments with the current SOTA detection methods on our dataset. The results reveal that existing models still struggle to achieve accurate and robust detection on our dataset. We hope that this new data set can promote research in the field of machine-generated content detection, fostering the responsible use of generative AI. The source code and datasets are available at https://github.com/ZhihaoZhang97/RU-AI.
Paper Structure (9 sections, 1 figure, 4 tables)

This paper contains 9 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. RU-AI Dataset
  3. Data Construction
  4. Data Processing
  5. Data Characteristics
  6. Proposed Baseline
  7. Implementation Details
  8. Results
  9. Conclusion

Figures (1)

  • Figure 1: Data collection process for all datasets. It shows the text collection (left), the image collection (middle), and the voice collection (right). Text is generated through rephrasing caption text. Images are created by referencing both caption text and real images. Voices are synthesized using caption text. All collected data is then matched with their original data pairs.