Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models

Kanishka Misra

TL;DR

minicons addresses the need for a standardized, open-source toolkit to perform both behavioral (predictions) and representational (activations) analyses of transformer-based language models. It introduces two core modules, scorer and cwe, built on PyTorch and HuggingFace, enabling efficient, batch-wise extraction of probabilities and contextualized embeddings with a flexible API and CLI. The authors validate the library through two case studies: tracking learning dynamics of BERT on 67 BLiMP phenomena and benchmarking zero-shot abductive reasoning across 23 pretrained LMs, revealing distinct acquisition trajectories and the challenges of unsupervised commonsense reasoning. The work contributes a practical, scalable framework for large-scale LM analysis and benchmarking, with implications for diagnostic research and ecosystem-wide evaluations within the HuggingFace community.

Abstract

We present minicons, an open source library that provides a standard API for researchers interested in conducting behavioral and representational analyses of transformer-based language models (LMs). Specifically, minicons enables researchers to apply analysis methods at two levels: (1) at the prediction level -- by providing functions to efficiently extract word/sentence level probabilities; and (2) at the representational level -- by also facilitating efficient extraction of word/phrase level vectors from one or more layers. In this paper, we describe the library and apply it to two motivating case studies: One focusing on the learning dynamics of the BERT architecture on relative grammatical judgments, and the other on benchmarking 23 different LMs on zero-shot abductive reasoning. minicons is available at https://github.com/kanishkamisra/minicons

minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models

TL;DR

minicons addresses the need for a standardized, open-source toolkit to perform both behavioral (predictions) and representational (activations) analyses of transformer-based language models. It introduces two core modules, scorer and cwe, built on PyTorch and HuggingFace, enabling efficient, batch-wise extraction of probabilities and contextualized embeddings with a flexible API and CLI. The authors validate the library through two case studies: tracking learning dynamics of BERT on 67 BLiMP phenomena and benchmarking zero-shot abductive reasoning across 23 pretrained LMs, revealing distinct acquisition trajectories and the challenges of unsupervised commonsense reasoning. The work contributes a practical, scalable framework for large-scale LM analysis and benchmarking, with implications for diagnostic research and ecosystem-wide evaluations within the HuggingFace community.

Abstract

We present minicons, an open source library that provides a standard API for researchers interested in conducting behavioral and representational analyses of transformer-based language models (LMs). Specifically, minicons enables researchers to apply analysis methods at two levels: (1) at the prediction level -- by providing functions to efficiently extract word/sentence level probabilities; and (2) at the representational level -- by also facilitating efficient extraction of word/phrase level vectors from one or more layers. In this paper, we describe the library and apply it to two motivating case studies: One focusing on the learning dynamics of the BERT architecture on relative grammatical judgments, and the other on benchmarking 23 different LMs on zero-shot abductive reasoning. minicons is available at https://github.com/kanishkamisra/minicons
Paper Structure (23 sections, 2 equations, 4 figures, 1 table)

This paper contains 23 sections, 2 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Overview of minicons
  3. Dependencies and Requirements
  4. Core Modules
  5. scorer
  6. cwe
  7. Relation to other Libraries
  8. Motivating Case Studies
  9. Learning Dynamics of Relative Linguistic Acceptability in LMs
  10. Data
  11. Models
  12. Method
  13. Analysis and Results
  14. Unsupervised Abductive Natural Language Inference
  15. Data
  16. ...and 8 more sections

Figures (4)

  • Figure 1: minicons facilitates analyses of pre-trained transformer-based language models at the prediction level through its scorer module and at the representational level through its cwe module. Detailed overview provided in \ref{['sec:overview']}.
  • Figure 2: Learning dynamics of MultiBerts sellam2021multiberts on various BLiMP phenomena plotted as accuracy over training time-steps. Error-bands represent 95% confidence intervals, while the dashed line represents performance of bert-base-uncaseddevlin-etal-2019-bert. Chance performance is 0.50 throughout.
  • Figure 3: Performance of the Unsupervised LM-scoring method on the $\alpha$NLI development set bhagavatula2019abductive. (a) Accuracy of each of the 23 models, arranged and colored based on the model family. Chance performance is 0.51 (dashed line), while current state of the art and human performance are 0.92 and 0.93, respectively (shown in dotted and solid lines, respectively). (b) Scatter plot of model accuracies versus parameter count (in millions; log-scaled). $R^2 = 0.48, p < .01$.
  • Figure 4: An example usage of the minicons CLI, showing results when GPT2 radford2019language is used to elicit word-level scores from the sentence the cat sat on the mat. Scores for first word are NaN as the first token is used to initiate the conditioning for the rest of the sentence.