Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Ordinality in Text Classification: A Comparative Study of Explicit and Implicit Techniques

Siva Rajesh Kasa, Aniket Goel, Karan Gupta, Sumegh Roychowdhury, Anish Bhanushali, Nikhil Pattisapu, Prasanna Srinivasa Murthy

TL;DR

This work tackles ordinal classification (OC) in NLP by comparing explicit ordinal-loss methods with implicit PLM-based approaches. It develops a unified framework around four properties—Proper Scoring Rule (PSR), Unimodality (UM), Convexity (Cx), and Ordinality (Ord)—to analyze explicit losses such as CE, OLL, SOFT, EMD, CORAL, WKL, and VS-SL, and introduces a hybrid Multi-task Log Loss (MLL) that blends CE and OLL via a tunable parameter $\lambda$ to balance nominal and ordinal metrics. On the implicit side, it investigates an encoder-based entailment-style approach using verbaliser templates and an image-like data-augmentation scheme, and a decoder-based generative approach with GPT-2 small and Llama-Adapter prompts, highlighting the role of informative versus uninformative verbalisers. Empirically, MLL offers balanced performance in high-data settings, ENT excels in few-shot scenarios with lower variance, and large decoder models (especially Llama-7B-Adapter) surpass others in full-data conditions, albeit with higher compute and hallucination risks. The paper provides practical recommendations for selecting OC strategies by data regime and emphasizes the importance of label semantics and unimodality as diagnostic signals for ordinal behavior.

Abstract

Ordinal Classification (OC) is a widely encountered challenge in Natural Language Processing (NLP), with applications in various domains such as sentiment analysis, rating prediction, and more. Previous approaches to tackle OC have primarily focused on modifying existing or creating novel loss functions that \textbf{explicitly} account for the ordinal nature of labels. However, with the advent of Pretrained Language Models (PLMs), it became possible to tackle ordinality through the \textbf{implicit} semantics of the labels as well. This paper provides a comprehensive theoretical and empirical examination of both these approaches. Furthermore, we also offer strategic recommendations regarding the most effective approach to adopt based on specific settings.

Exploring Ordinality in Text Classification: A Comparative Study of Explicit and Implicit Techniques

TL;DR

This work tackles ordinal classification (OC) in NLP by comparing explicit ordinal-loss methods with implicit PLM-based approaches. It develops a unified framework around four properties—Proper Scoring Rule (PSR), Unimodality (UM), Convexity (Cx), and Ordinality (Ord)—to analyze explicit losses such as CE, OLL, SOFT, EMD, CORAL, WKL, and VS-SL, and introduces a hybrid Multi-task Log Loss (MLL) that blends CE and OLL via a tunable parameter to balance nominal and ordinal metrics. On the implicit side, it investigates an encoder-based entailment-style approach using verbaliser templates and an image-like data-augmentation scheme, and a decoder-based generative approach with GPT-2 small and Llama-Adapter prompts, highlighting the role of informative versus uninformative verbalisers. Empirically, MLL offers balanced performance in high-data settings, ENT excels in few-shot scenarios with lower variance, and large decoder models (especially Llama-7B-Adapter) surpass others in full-data conditions, albeit with higher compute and hallucination risks. The paper provides practical recommendations for selecting OC strategies by data regime and emphasizes the importance of label semantics and unimodality as diagnostic signals for ordinal behavior.

Abstract

Ordinal Classification (OC) is a widely encountered challenge in Natural Language Processing (NLP), with applications in various domains such as sentiment analysis, rating prediction, and more. Previous approaches to tackle OC have primarily focused on modifying existing or creating novel loss functions that \textbf{explicitly} account for the ordinal nature of labels. However, with the advent of Pretrained Language Models (PLMs), it became possible to tackle ordinality through the \textbf{implicit} semantics of the labels as well. This paper provides a comprehensive theoretical and empirical examination of both these approaches. Furthermore, we also offer strategic recommendations regarding the most effective approach to adopt based on specific settings.
Paper Structure (24 sections, 24 equations, 4 figures, 9 tables)

This paper contains 24 sections, 24 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Explicit approach: Loss-functions and Analysis
  3. Desirable Properties of Losses in OC
  4. Widely used Loss Functions in OC
  5. Implicit approach: Entailment-style Encoder Models
  6. Implicit approach: Generative Models
  7. Results and Discussion
  8. Conclusion
  9. Limitations
  10. Comparison of different loss based approaches
  11. Datasets
  12. SNLI
  13. Amazon Reviews
  14. SST5
  15. Metrics
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Classifier output on the top does not satisfy UM as the probabilities do not decrease monotonically on either side of the max probability label (label: average); whereas the classifier output on the bottom satisfies UM
  • Figure 2: [Best viewed in color] (1) Explicit Approach: We explicitly impose ordinality in the predictions by designing losses like MLL, OLL, etc. (2) Implicit Approach: In this entailment technique, ordinality is implicitly enforced by passing the label semantic information in the input itself. (3) Here we use a decoder LM to auto-regressively predict the label in natural language format given the input sentence. (Notation:$PLM_{E}$: Encoder PLM, $PLM_{D}$: Decoder PLM)
  • Figure 3: Effect of informative vs uninformative prompts in Implicit approaches on the nominal (F1) and ordinal metrics (MAE, MSE, OB1) on SNLI (top), SST5 (middle) and Amazon Reviews (bottom) datasets. Results averaged across 5 random seeds. (Notation: _U refers to the un-informative verbaliser variant. Weighted-F1 and OB1: $\uparrow$ better, MAE and MSE: $\downarrow$ better.)
  • Figure 4: % samples following UM property.