When in Doubt, Deliberate: Confidence-Based Routing to Expert Debate for Sexism Detection

TL;DR

The paper tackles the challenge of detecting sexism in nuanced online content by addressing data scarcity, label noise, and conceptual ambiguity through a two-stage framework that combines targeted supervision with confidence-based routing to a Collaborative Expert Judgment module.A domain-tuned classifier is trained with class-balanced focal loss, class-aware batching, and post-hoc calibration, then inference routes uncertain cases to a multi-persona debate among experts whose reasoning is synthesized by a judge to produce the final decision.Empirical evaluations on EXIST 2025 and EDOS benchmarks demonstrate state-of-the-art performance, with notable improvements for underrepresented and ambiguous classes, and ablations confirm the value of multi-persona deliberation and careful prompt design.The work contributes a scalable, interpretable approach that integrates robust task-specific training with structured, multi-perspective reasoning, offering practical benefits for moderation systems facing linguistic and cultural variability.

Abstract

Sexist content online increasingly appears in subtle, context-dependent forms that evade traditional detection methods. Its interpretation often depends on overlapping linguistic, psychological, legal, and cultural dimensions, which produce mixed and sometimes contradictory signals, even in annotated datasets. These inconsistencies, combined with label scarcity and class imbalance, result in unstable decision boundaries and cause fine-tuned models to overlook subtler, underrepresented forms of harm. Together, these limitations point to the need for a design that explicitly addresses the combined effects of (i) underrepresentation, (ii) noise, and (iii) conceptual ambiguity in both data and model predictions. To address these challenges, we propose a two-stage framework that unifies (i) targeted training procedures to adapt supervision to scarce and noisy data with (ii) selective, reasoning-based inference to handle ambiguous or borderline cases. Our training setup applies class-balanced focal loss, class-aware batching, and post-hoc threshold calibration to mitigate label imbalance and noisy supervision. At inference time, a dynamic routing mechanism classifies high-confidence cases directly and escalates uncertain instances to a novel \textit{Collaborative Expert Judgment} (CEJ) module, which prompts multiple personas and consolidates their reasoning through a judge model. Our approach achieves state-of-the-art results across several benchmarks, with a +2.72\% improvement in F1 on the EXIST 2025 Task 1.1, and a gains of +4.48\% and +1.30\% on the EDOS Tasks A and B, respectively.

Figures (9)

• Figure 1: Overview of the proposed routing mechanism. Predictions with low confidence from the fine-tuned model are escalated to a Collaborative Expert Judgment (CEJ) module, where multiple LLM-based personas engage in structured debate, before the final decision.
• Figure 2: CEJ framework, where a collaborative reasoning discussion among personas with refinement is generated before the judge model makes a definitive decision.
• Figure 3: Performance gain after the debate phase.
• Figure A1: An example output from the initial classification stage of the CEJ framework.
• Figure A2: An example output from the structured debate stage of the CEJ framework.