Style Extraction on Text Embeddings Using VAE and Parallel Dataset

TL;DR

This work targets the problem of quantifying stylistic differences across Bible translations by treating style as a separable component in text embeddings. It employs high-dimensional sentence embeddings (Paradigm: $d=1536$) and trains a Variational Autoencoder on the vector differences between the KJV and ASV translations to model ASV-style, using the $L_2$ reconstruction error to detect style deviations. By evaluating across multiple translations with Fisher’s Linear Discriminant and comparing context-subtracted versus non-subtracted inputs, the study demonstrates that each translation exhibits a distinct stylistic distribution and that the VAE can differentiate ASV from others with an accuracy of about $84.7 ext{%}$, while highlighting limitations in capturing multi-style relationships. The findings suggest broad potential for AI-assisted stylistic analysis and controlled text generation, and indicate directions for extending the methodology to other parallel corpora and domains with more nuanced style interactions, potentially requiring more sophisticated models beyond a single-style VAE framework.

Abstract

This study investigates the stylistic differences among various Bible translations using a Variational Autoencoder (VAE) model. By embedding textual data into high-dimensional vectors, the study aims to detect and analyze stylistic variations between translations, with a specific focus on distinguishing the American Standard Version (ASV) from other translations. The results demonstrate that each translation exhibits a unique stylistic distribution, which can be effectively identified using the VAE model. These findings suggest that the VAE model is proficient in capturing and differentiating textual styles, although it is primarily optimized for distinguishing a single style. The study highlights the model's potential for broader applications in AI-based text generation and stylistic analysis, while also acknowledging the need for further model refinement to address the complexity of multi-dimensional stylistic relationships. Future research could extend this methodology to other text domains, offering deeper insights into the stylistic features embedded within various types of textual data.

Figures (6)

• Figure 1: A schematic illustration of the VAE model. The encoder receives a 1,536-dimensional original (sentence embedding) vector as input and outputs a feature vector of the feature dimension. The decoder takes the feature vector of the feature dimension as input and outputs a 1,536-dimensional reconstructed vector.
• Figure 2: Test set loss during training. The x-axis represents the number of epochs, and the y-axis represents the mean error. The hyperparameters of each model are as follows: starting from left the 1st, 2nd, and 3rd columns represent feature dimensions of 8, 64, and 256, respectively, and the starting from top 1st, 2nd, and 3rd rows represent 1, 3, and 6 hidden layers, respectively.
• Figure 3: L2 error distribution on ASV, NET, ASVS, Coverdale, Geneva, and KJV Strongs. The x-axis represents the L2 error between the original and reconstructed sentence vector, and the y-axis represents the distribution density. The hyperparameters of each model are as follows: starting from left the 1st, 2nd, and 3rd columns represent feature dimensions of 8, 64, and 256, respectively, and starting from top the 1st, 2nd, and 3rd rows represent 1, 3, and 6 hidden layers, respectively.
• Figure 4: (Left) Minimum and (Right) Maximum of FLD between ASV and other 5 anomaly datasets (NET, ASVS, Coverdale, Geneva, and KJV Strongs). A higher minimum FLD indicates better differentiation between ASV and anomaly L2 error distributions.
• Figure 5: L2 error distribution on ASV, NET, ASVS, Coverdale, Geneva, and KJV Strongs, without parallel sentence (KJV) subtraction. The x-axis represents the L2 error between the original and reconstructed sentence vector, and the y-axis represents the distribution density. The hyperparameters of each model are as follows: starting from left the 1st, 2nd, and 3rd columns represent feature dimensions of 8, 64, and 256, respectively, and the starting from top 1st, 2nd, and 3rd rows represent 1, 3, and 6 hidden layers, respectively.