Experimentation in Content Moderation using RWKV

TL;DR

A novel dataset specifically designed for distillation into smaller models is introduced, enhancing content moderation practices and demonstrating RWKV's capability in improving the accuracy and efficiency of content moderation systems but also paves the way for developing more compact, resource-efficient models in this domain.

Abstract

This paper investigates the RWKV model's efficacy in content moderation through targeted experimentation. We introduce a novel dataset specifically designed for distillation into smaller models, enhancing content moderation practices. This comprehensive dataset encompasses images, videos, sounds, and text data that present societal challenges. Leveraging advanced Large Language Models (LLMs), we generated an extensive set of responses -- 558,958 for text and 83,625 for images -- to train and refine content moderation systems. Our core experimentation involved fine-tuning the RWKV model, capitalizing on its CPU-efficient architecture to address large-scale content moderation tasks. By highlighting the dataset's potential for knowledge distillation, this study not only demonstrates RWKV's capability in improving the accuracy and efficiency of content moderation systems but also paves the way for developing more compact, resource-efficient models in this domain. Datasets and models can be found in HuggingFace: https://huggingface.co/modrwkv

Figures (5)

• Figure 1: Synthetic dataset generation process
• Figure 2: Example conversation with Mod-RWKV and Mod-VisualRWKV
• Figure 3: Image Dataset Distribution
• Figure 3: Accuracies for various models based on 1000 sample evaluations
• Figure 4: Text Dataset Distribution