Towards Trustworthy Multimodal Moderation via Policy-Aligned Reasoning and Hierarchical Labeling

TL;DR

Hi-Guard tackles the misalignment, opacity, and fine-grained challenges of multimodal content moderation by deploying a policy-aligned, two-stage cascade with a hierarchical taxonomy (Domain → Topic → Subtype → Behavior). It grounds decisions in explicit moderation rules through structured prompts, incorporates chain-of-thought reasoning, and optimizes with Group Relative Policy Optimization (GRPO) using a multi-level soft-margin reward. The four-level taxonomy enables path-based classification with reduced search space and improved generalization, while the reward design penalizes sibling confusions and emphasizes deeper, policy-relevant distinctions. Empirical results on offline datasets and online deployment show improved accuracy, interpretability, and efficiency, along with substantial reductions in human moderation workload. This approach demonstrates a scalable, transparent framework for policy-aligned content safety in real-world platforms, with practical impact on safety and governance processes.

Abstract

Social platforms have revolutionized information sharing, but also accelerated the dissemination of harmful and policy-violating content. To ensure safety and compliance at scale, moderation systems must go beyond efficiency and offer accuracy and interpretability. However, current approaches largely rely on noisy, label-driven learning, lacking alignment with moderation rules and producing opaque decisions that hinder human review. Therefore, we propose Hierarchical Guard (Hi-Guard), a multimodal moderation framework that introduces a new policy-aligned decision paradigm. The term "Hierarchical" reflects two key aspects of our system design: (1) a hierarchical moderation pipeline, where a lightweight binary model first filters safe content and a stronger model handles fine-grained risk classification; and (2) a hierarchical taxonomy in the second stage, where the model performs path-based classification over a hierarchical taxonomy ranging from coarse to fine-grained levels. To ensure alignment with evolving moderation policies, Hi-Guard directly incorporates rule definitions into the model prompt. To further enhance structured prediction and reasoning, we introduce a multi-level soft-margin reward and optimize with Group Relative Policy Optimization (GRPO), penalizing semantically adjacent misclassifications and improving explanation quality. Extensive experiments and real-world deployment demonstrate that Hi-Guard achieves superior classification accuracy, generalization, and interpretability, paving the way toward scalable, transparent, and trustworthy content safety systems. Code is available at: https://github.com/lianqi1008/Hi-Guard.

Figures (9)

• Figure 1: Moderation system architecture. The term “layer” here refers to functional stages in the moderation pipeline. Stages enclosed by solid lines are inherent components of the system, while dashed boxes indicate the stage we aim to optimize.
• Figure 2: The framework of our proposed Hi-Guard. “Hierarchical” refers to both the two-stage moderation pipeline and the multi-level taxonomy used for path-based classification. The binary guard recognizes the input notes and captures the potentially risky notes, the hierarchical guard then accepts them and platform rules as the input, with format and soft-margin accuracy reward, driving the model to learn semantic structure among labels and search for the correct reasoning path.
• Figure 3: Illustration of Hi-Guard’s structural benefits. (a) The hierarchical taxonomy reduces the prediction space by limiting candidate labels at each level. (b) Sibling categories under the same level are more easily confused. Hi-Guard encourages the model to separate such categories more clearly.
• Figure 4: Illustration of the structured prompt used in Hi-Guard.
• Figure 5: Human preference distribution over CoT explanations from three models (RLVR, Hi-Guard w/o SMR, Hi-Guard) across 170 evaluation samples. Bars represent the average proportion of times each model is rated as best, neutral, or worst by five professional moderators.