TeSent: A Benchmark Dataset for Fairness-aware Explainable Sentiment Classification in Telugu

TL;DR

TeSent addresses the scarcity of Telugu sentiment benchmarks by delivering a large-scale, human-annotated dataset with explicit rationales and a fairness evaluation corpus (TeEEC). It probes alignment between human reasoning and model explanations by training five transformer models with and without rationales and by applying a suite of post-hoc explainers, observing improved plausibility and often better accuracy when rationales are used. However, alignment-oriented training does not guarantee fairness improvements, underscoring the need for explicit fairness constraints in low-resource languages. The work provides publicly available resources and a rigorous evaluation framework to advance fair, interpretable NLP for Telugu and other Indic languages.

Abstract

In the Indian subcontinent, Telugu, one of India's six classical languages, is the most widely spoken Dravidian Language. Despite its 96 million speaker base worldwide, Telugu remains underrepresented in the global NLP and Machine Learning landscape, mainly due to lack of high-quality annotated resources. This work introduces TeSent, a comprehensive benchmark dataset for sentiment classification, a key text classification problem, in Telugu. TeSent not only provides ground truth labels for the sentences, but also supplements with provisions for evaluating explainability and fairness, two critical requirements in modern-day machine learning tasks. We scraped Telugu texts covering multiple domains from various social media platforms, news websites and web-blogs to preprocess and generate 21,119 sentences, and developed a custom-built annotation platform and a carefully crafted annotation protocol for collecting the ground truth labels along with their human-annotated rationales. We then fine-tuned several SOTA pre-trained models in two ways: with rationales, and without rationales. Further, we provide a detailed plausibility and faithfulness evaluation suite, which exploits the rationales, for six widely used post-hoc explainers applied on the trained models. Lastly, we curate TeEEC, Equity Evaluation Corpus in Telugu, a corpus to evaluate fairness of Telugu sentiment and emotion related NLP tasks, and provide a fairness evaluation suite for the trained classifier models. Our experimental results suggest that training with human rationales improves model accuracy and models' alignment with human reasoning, but does not necessarily reduce bias.

Figures (6)

• Figure 1: Distribution of Agreement Levels
• Figure 2: Data Split for TeSent
• Figure 3: Word Count Frequency for TeSent
• Figure 4: Category Wordcloud for TeSent
• Figure 5: Annotation Distribution