Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Narrative Clustering in Large Language Models: A Layerwise Analysis of BERT

Awritrojit Banerjee, Achim Schilling, Patrick Krauss

TL;DR

The paper investigates how BERT encodes narrative content versus authorial style in its layerwise activations. It uses 1000 narratives generated via neural style transfer and analyzes per-layer [CLS] embeddings with PCA and MDS, complemented by the GDV metric. Results reveal strong content-based clustering in later layers and minimal clustering by author, indicating semantic emphasis over stylistic features. This supports the view that encoder-only transformers implement hierarchical semantic abstractions and provides a bridge between AI representations and cognitive neuroscience.

Abstract

This study investigates the internal mechanisms of BERT, a transformer-based large language model, with a focus on its ability to cluster narrative content and authorial style across its layers. Using a dataset of narratives developed via GPT-4, featuring diverse semantic content and stylistic variations, we analyze BERT's layerwise activations to uncover patterns of localized neural processing. Through dimensionality reduction techniques such as Principal Component Analysis (PCA) and Multidimensional Scaling (MDS), we reveal that BERT exhibits strong clustering based on narrative content in its later layers, with progressively compact and distinct clusters. While strong stylistic clustering might occur when narratives are rephrased into different text types (e.g., fables, sci-fi, kids' stories), minimal clustering is observed for authorial style specific to individual writers. These findings highlight BERT's prioritization of semantic content over stylistic features, offering insights into its representational capabilities and processing hierarchy. This study contributes to understanding how transformer models like BERT encode linguistic information, paving the way for future interdisciplinary research in artificial intelligence and cognitive neuroscience.

Exploring Narrative Clustering in Large Language Models: A Layerwise Analysis of BERT

TL;DR

The paper investigates how BERT encodes narrative content versus authorial style in its layerwise activations. It uses 1000 narratives generated via neural style transfer and analyzes per-layer [CLS] embeddings with PCA and MDS, complemented by the GDV metric. Results reveal strong content-based clustering in later layers and minimal clustering by author, indicating semantic emphasis over stylistic features. This supports the view that encoder-only transformers implement hierarchical semantic abstractions and provides a bridge between AI representations and cognitive neuroscience.

Abstract

This study investigates the internal mechanisms of BERT, a transformer-based large language model, with a focus on its ability to cluster narrative content and authorial style across its layers. Using a dataset of narratives developed via GPT-4, featuring diverse semantic content and stylistic variations, we analyze BERT's layerwise activations to uncover patterns of localized neural processing. Through dimensionality reduction techniques such as Principal Component Analysis (PCA) and Multidimensional Scaling (MDS), we reveal that BERT exhibits strong clustering based on narrative content in its later layers, with progressively compact and distinct clusters. While strong stylistic clustering might occur when narratives are rephrased into different text types (e.g., fables, sci-fi, kids' stories), minimal clustering is observed for authorial style specific to individual writers. These findings highlight BERT's prioritization of semantic content over stylistic features, offering insights into its representational capabilities and processing hierarchy. This study contributes to understanding how transformer models like BERT encode linguistic information, paving the way for future interdisciplinary research in artificial intelligence and cognitive neuroscience.
Paper Structure (16 sections, 4 equations, 5 figures)

This paper contains 16 sections, 4 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Methodology
  3. Dataset Construction
  4. Selection of Authors and Texts
  5. Neural Style Transfer
  6. BERT: Representation Extraction
  7. BERT as an Encoder-Only Transformer
  8. Tokenization and Embedding Retrieval
  9. Dimensionality Reduction and Projection
  10. Principal Component Analysis (PCA)
  11. Multidimensional Scaling (MDS)
  12. Degree of Clustering
  13. Summary
  14. Results
  15. Conclusion
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Layerwise Neural Activations in BERT Projected using MDS when Clustered Based on Authors
  • Figure 2: Layerwise Neural Activations in BERT Projected using PCA when Clustered Based on Authors
  • Figure 3: Layerwise Neural Activations in BERT Projected using MDS when Clustered Based on the Narratives
  • Figure 4: Layerwise Neural Activations in BERT Projected using PCA when Clustered Based on the Narratives
  • Figure 5: Layerwise Trends in GDV Illustrating Degree of Clustering (lower is better)