Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simple Visual Artifact Detection in Sora-Generated Videos

Misora Sugiyama, Hirokatsu Kataoka

TL;DR

This paper tackles safety and reliability concerns in VidLLMs by analyzing artifacts in Sora-generated videos using a four-label taxonomy. It develops a multi-label classification framework tested on 300 frames from 15 videos, comparing 2D CNN architectures (ResNet-50, EfficientNet-B3/B4, ViT-Base) for artifact detection. ResNet-50 yields the best overall performance with a mean per-label accuracy of 94.14%, while Grad-CAM analysis provides insights into model focus and interpretability. The study contributes data and methodological groundwork for video-quality evaluation and artifact-based analysis, with suggested future work on larger datasets and temporal modeling to enhance robustness against motion-related artifacts.</mean></p>

Abstract

The December 2024 release of OpenAI's Sora, a powerful video generation model driven by natural language prompts, highlights a growing convergence between large language models (LLMs) and video synthesis. As these multimodal systems evolve into video-enabled LLMs (VidLLMs), capable of interpreting, generating, and interacting with visual content, understanding their limitations and ensuring their safe deployment becomes essential. This study investigates visual artifacts frequently found and reported in Sora-generated videos, which can compromise quality, mislead viewers, or propagate disinformation. We propose a multi-label classification framework targeting four common artifact label types: label 1: boundary / edge defects, label 2: texture / noise issues, label 3: movement / joint anomalies, and label 4: object mismatches / disappearances. Using a dataset of 300 manually annotated frames extracted from 15 Sora-generated videos, we trained multiple 2D CNN architectures (ResNet-50, EfficientNet-B3 / B4, ViT-Base). The best-performing model trained by ResNet-50 achieved an average multi-label classification accuracy of 94.14%. This work supports the broader development of VidLLMs by contributing to (1) the creation of datasets for video quality evaluation, (2) interpretable artifact-based analysis beyond language metrics, and (3) the identification of visual risks relevant to factuality and safety.

Simple Visual Artifact Detection in Sora-Generated Videos

TL;DR

This paper tackles safety and reliability concerns in VidLLMs by analyzing artifacts in Sora-generated videos using a four-label taxonomy. It develops a multi-label classification framework tested on 300 frames from 15 videos, comparing 2D CNN architectures (ResNet-50, EfficientNet-B3/B4, ViT-Base) for artifact detection. ResNet-50 yields the best overall performance with a mean per-label accuracy of 94.14%, while Grad-CAM analysis provides insights into model focus and interpretability. The study contributes data and methodological groundwork for video-quality evaluation and artifact-based analysis, with suggested future work on larger datasets and temporal modeling to enhance robustness against motion-related artifacts.</mean></p>

Abstract

The December 2024 release of OpenAI's Sora, a powerful video generation model driven by natural language prompts, highlights a growing convergence between large language models (LLMs) and video synthesis. As these multimodal systems evolve into video-enabled LLMs (VidLLMs), capable of interpreting, generating, and interacting with visual content, understanding their limitations and ensuring their safe deployment becomes essential. This study investigates visual artifacts frequently found and reported in Sora-generated videos, which can compromise quality, mislead viewers, or propagate disinformation. We propose a multi-label classification framework targeting four common artifact label types: label 1: boundary / edge defects, label 2: texture / noise issues, label 3: movement / joint anomalies, and label 4: object mismatches / disappearances. Using a dataset of 300 manually annotated frames extracted from 15 Sora-generated videos, we trained multiple 2D CNN architectures (ResNet-50, EfficientNet-B3 / B4, ViT-Base). The best-performing model trained by ResNet-50 achieved an average multi-label classification accuracy of 94.14%. This work supports the broader development of VidLLMs by contributing to (1) the creation of datasets for video quality evaluation, (2) interpretable artifact-based analysis beyond language metrics, and (3) the identification of visual risks relevant to factuality and safety.
Paper Structure (11 sections, 3 figures, 2 tables)

This paper contains 11 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Materials and Methods
  3. Data Collection and Annotation
  4. Model Selection and Training
  5. Evaluation Metrics
  6. Experiments
  7. Artifact Frequency
  8. Model Performance
  9. Grad-CAM Observations
  10. Limitations
  11. Conclusion

Figures (3)

  • Figure 1: Examples of four artifact categories: top left “boundary/edge defects,” top right “texture/noise issues,” bottom left “motion/joint anomalies,” bottom right “object mismatches / disappearances”
  • Figure 2: The training and validation loss curves.
  • Figure 3: Difference between where the model and human focus their attention. Above: where the model makes decisions (Grad-CAM) / Below: where the human pays attention.