Ethic-BERT: An Enhanced Deep Learning Model for Ethical and Non-Ethical Content Classification

TL;DR

Ethic-BERT addresses the challenge of nuanced ethical content classification by fine-tuning a BERT base cased model on the ETHICS dataset across four domains (Commonsense, Justice, Virtue, Deontology). The approach combines bias-aware preprocessing, full unfreezing of model layers, gradient accumulation, and adaptive learning-rate scheduling, evaluated against an adversarial Hard Test to measure robustness. Results show superior standard-test accuracy (average 82.32%) and notable gains on Hard Test (63.18%), outperforming BERT-base, RoBERTa-large, and ALBERT-xxlarge, particularly in Justice and Virtue. The work highlights the value of comprehensive fine-tuning and robust preprocessing for reliable ethical reasoning, while outlining future directions such as richer context, external knowledge, and multitask learning to close remaining gaps in Commonsense and Deontology.

Abstract

Developing AI systems capable of nuanced ethical reasoning is critical as they increasingly influence human decisions, yet existing models often rely on superficial correlations rather than principled moral understanding. This paper introduces Ethic-BERT, a BERT-based model for ethical content classification across four domains: Commonsense, Justice, Virtue, and Deontology. Leveraging the ETHICS dataset, our approach integrates robust preprocessing to address vocabulary sparsity and contextual ambiguities, alongside advanced fine-tuning strategies like full model unfreezing, gradient accumulation, and adaptive learning rate scheduling. To evaluate robustness, we employ an adversarially filtered "Hard Test" split, isolating complex ethical dilemmas. Experimental results demonstrate Ethic-BERT's superiority over baseline models, achieving 82.32% average accuracy on the standard test, with notable improvements in Justice and Virtue. In addition, the proposed Ethic-BERT attains 15.28% average accuracy improvement in the HardTest. These findings contribute to performance improvement and reliable decision-making using bias-aware preprocessing and proposed enhanced AI model.

Figures (5)

• Figure 1: Overview of the methodology for ethical reasoning
• Figure 2: Fine tuning of BERT for ethical reasoning
• Figure 3: Confusion matrix of Test dataset.
• Figure 4: Confusion matrix of Hard Test dataset.
• Figure 5: Training vs validation loss and accuracy curves on training dataset.