An Information-Theoretic Approach to Analyze NLP Classification Tasks

TL;DR

An information-theoretic framework is proposed to quantify how input elements and their semantic versus linguistic components shape output distributions in NLP classification, formalized with $I(Y; \mathbf{X})$ and a decomposition into $I(Y;X_j^{(s)})$ and $I(Y;X_j|X_j^{(s)})$. The framework is instantiated on multiple-choice reading comprehension (MCRC) and sentiment classification (SC), using paraphrase-based linguistic realizations to separately assess semantic and linguistic influence. Key findings show context and its semantic content strongly influence MCRC outputs, while semantic content dominates SC across datasets, with measurable but smaller linguistic effects; the approach also enables calibration and data-complexity analyses. The work provides a practical diagnostic tool for designing datasets and evaluating input factors, and the authors release their framework and experiments at GitHub for broader use.

Abstract

Understanding the importance of the inputs on the output is useful across many tasks. This work provides an information-theoretic framework to analyse the influence of inputs for text classification tasks. Natural language processing (NLP) tasks take either a single element input or multiple element inputs to predict an output variable, where an element is a block of text. Each text element has two components: an associated semantic meaning and a linguistic realization. Multiple-choice reading comprehension (MCRC) and sentiment classification (SC) are selected to showcase the framework. For MCRC, it is found that the context influence on the output compared to the question influence reduces on more challenging datasets. In particular, more challenging contexts allow a greater variation in complexity of questions. Hence, test creators need to carefully consider the choice of the context when designing multiple-choice questions for assessment. For SC, it is found the semantic meaning of the input text dominates (above 80\% for all datasets considered) compared to its linguistic realisation when determining the sentiment. The framework is made available at: https://github.com/WangLuran/nlp-element-influence

Figures (10)

• Figure 1: Data generation for multiple-choice reading comprehension for the context (blue) and question (purple) respectively.
• Figure 2: Architectures for multiple-choice reading comprehension with context, $c$, question, $q$ and options, $o$.
• Figure 3: Architecture for MC question complexity classifier with context, $c$, question, $q$ and options, $\{o\}$.
• Figure 4: Normalized ranks (rank / total examples) of complexity scores for each complexity level using three complexity evaluators: context, context-question and standard.
• Figure 5: The relative question influence changes with the subset chosen by the rank of context complexity in all three datasets (left) and in RACE++ only (right). 0.2 in x-axis means we leave contexts with top 20% context complexity as the subset.