Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policies

Peiran Wang, Qiyu Li, Longxuan Yu, Ziyao Wang, Ang Li, Haojian Jin

Abstract

We present Moderator, a policy-based model management system that allows administrators to specify fine-grained content moderation policies and modify the weights of a text-to-image (TTI) model to make it significantly more challenging for users to produce images that violate the policies. In contrast to existing general-purpose model editing techniques, which unlearn concepts without considering the associated contexts, Moderator allows admins to specify what content should be moderated, under which context, how it should be moderated, and why moderation is necessary. Given a set of policies, Moderator first prompts the original model to generate images that need to be moderated, then uses these self-generated images to reverse fine-tune the model to compute task vectors for moderation and finally negates the original model with the task vectors to decrease its performance in generating moderated content. We evaluated Moderator with 14 participants to play the role of admins and found they could quickly learn and author policies to pass unit tests in approximately 2.29 policy iterations. Our experiment with 32 stable diffusion users suggested that Moderator can prevent 65% of users from generating moderated content under 15 attempts and require the remaining users an average of 8.3 times more attempts to generate undesired content.

Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policies

Abstract

We present Moderator, a policy-based model management system that allows administrators to specify fine-grained content moderation policies and modify the weights of a text-to-image (TTI) model to make it significantly more challenging for users to produce images that violate the policies. In contrast to existing general-purpose model editing techniques, which unlearn concepts without considering the associated contexts, Moderator allows admins to specify what content should be moderated, under which context, how it should be moderated, and why moderation is necessary. Given a set of policies, Moderator first prompts the original model to generate images that need to be moderated, then uses these self-generated images to reverse fine-tune the model to compute task vectors for moderation and finally negates the original model with the task vectors to decrease its performance in generating moderated content. We evaluated Moderator with 14 participants to play the role of admins and found they could quickly learn and author policies to pass unit tests in approximately 2.29 policy iterations. Our experiment with 32 stable diffusion users suggested that Moderator can prevent 65% of users from generating moderated content under 15 attempts and require the remaining users an average of 8.3 times more attempts to generate undesired content.
Paper Structure (28 sections, 17 figures, 5 tables)

This paper contains 28 sections, 17 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Threat model
  3. Background
  4. Understanding Content Moderation Needs in TTI Models
  5. Policy Design
  6. Policy Development
  7. Semi-structured Context-based Policy
  8. Running Policy Examples
  9. Policy-based Model Transformation
  10. System Primitive: Self-reverse Fine-tuning
  11. Fine-grained Moderation
  12. Moderating Diverse Prompts
  13. Moderation Methods
  14. Multi-policy Interference
  15. Advanced Policies
  16. ...and 13 more sections

Figures (17)

  • Figure 1: Given (a) a policy that moderates "Tom Hanks advertises McDonald" fakeTomHanks:online, Moderator can (b, c, d) prevent the model from generating images that depict "Tom Hanks advertises McDonald" while (e, f, g) preserving the model's ability to generate normal Tom Hanks images.
  • Figure 2: Self-reverse fine-tuning has three steps: (1) generating undesired images using the policy, (2) fine-tuning with undesired images and extracting the task vector that represents the mapping relation between the input prompt and output images, and (3) negating the original model with the task vectors to decrease its performance in generating moderated content.
  • Figure 3: Moderator uses an LLM to (1) expand the policy coverage and (2) generate high-quality prompts.
  • Figure 4: Moderator allows admins to configure a (a) replace policy to replace Disneyland figures with a regular mouse. Through automatic policy expansion, Moderator also moderates relevant concepts (e.g., (b, c, d) Donald Duck and (e, f, g) Mickey Mouse) under "Disneyland figures," even though the admins did not mention them explicitly in the policy.
  • Figure 5: Moderator replaces content by reverse fine-tuning a special dataset, in which we map (a) the original prompt to (d) the output from (b) modified prompts.
  • ...and 12 more figures