So-Fake: Benchmarking and Explaining Social Media Image Forgery Detection

TL;DR

So-Fake tackles the challenge of social-media image forgery by introducing a large-scale, socially realistic benchmark (So-Fake-Set and So-Fake-OOD) and a unified RL-based framework (So-Fake-R1) that jointly performs detection, localization, and explanation. By combining a diverse training/validation set with a challenging out-of-domain testbed built from Reddit content and unseen commercial generators, the authors demonstrate robust cross-domain generalization. So-Fake-R1 uses a two-stage training regime and Group Relative Policy Optimization to balance multi-task objectives and produce interpretable outputs, achieving state-of-the-art results on both in-domain and OOD evaluations. The work provides a practical foundation for advancing trustworthy forgery detection on social media and emphasizes the importance of integrated detection, localization, and explanation for user trust. Limitations include the still-evolving generation landscape, computational demands, and the focus on still images, suggesting future work in video and multimodal forgery analysis.

Abstract

Recent advances in AI-powered generative models have enabled the creation of increasingly realistic synthetic images, posing significant risks to information integrity and public trust on social media platforms. While robust detection frameworks and diverse, large-scale datasets are essential to mitigate these risks, existing academic efforts remain limited in scope: current datasets lack the diversity, scale, and realism required for social media contexts, while detection methods struggle with generalization to unseen generative technologies. To bridge this gap, we introduce So-Fake-Set, a comprehensive social media-oriented dataset with over 2 million high-quality images, diverse generative sources, and photorealistic imagery synthesized using 35 state-of-the-art generative models. To rigorously evaluate cross-domain robustness, we establish a novel and large-scale (100K) out-of-domain benchmark (So-Fake-OOD) featuring synthetic imagery from commercial models explicitly excluded from the training distribution, creating a realistic testbed for evaluating real-world performance. Leveraging these resources, we present So-Fake-R1, an advanced vision-language framework that employs reinforcement learning for highly accurate forgery detection, precise localization, and explainable inference through interpretable visual rationales. Extensive experiments show that So-Fake-R1 outperforms the second-best method, with a 1.3% gain in detection accuracy and a 4.5% increase in localization IoU. By integrating a scalable dataset, a challenging OOD benchmark, and an advanced detection framework, this work establishes a new foundation for social media-centric forgery detection research. The code, models, and datasets will be released publicly.

Figures (24)

• Figure 1: (a) Overview. So-Fake comprises So-Fake-Set (train/val) and So-Fake-OOD (test), which together enable evaluation of detection, localization, and explanation with So-Fake-R1. (b) Illustrative Example. A real image from the subreddit pics is captioned by an LLM, combined with Language SAM and an inpainting model to produce tampered samples. So-Fake-R1 then analyzes the manipulated image and outputs the class label, localized region, and an interpretable rationale.
• Figure 2: (a) Category distribution of So-Fake across 12 semantic classes. (b) Generative methods and examples for each split–task combination.
• Figure 3: (a) Data collection sources of So-Fake-Set and So-Fake-OOD with representative examples and statistics. (b) Data generation pipelines for full synthetic and tampered images.
• Figure 4: (a) Training pipeline with SFT and GRPO. (b) Inference pipeline producing detection, localization, and explanation outputs.
• Figure 5: Cross-generator domain generalization matrix using CNNSpot. Rows indicate training generators and columns indicate test generators, with So-Fake-Set (left) and So-Fake-OOD (right).