Scalable Multi-phase Word Embedding Using Conjunctive Propositional Clauses

TL;DR

Problem addressed: scalable, interpretable word embeddings for NLP using symbolic propositional logic. Approach: a two-phase TM-AE on a Coalesced Tsetlin Machine where Phase 1 extracts word level knowledge from the vocabulary of size $|V|$ with a context window of size $a$, and Phase 2 builds embeddings for word sequences via clause-to-clause encoding using the Phase 1 knowledge. Contributions: Phase 1 produces interpretable clauses describing each word, Phase 2 yields embeddings for word vectors usable for downstream tasks and data augmentation, with an end-to-end transparent architecture. Findings and impact: Phase 2 achieves strong Cosine similarity on RG65 ($0.91$) and competitive sentiment analysis results on IMDB, demonstrating scalable and interpretable embeddings that complement deep contextual models. Significance: provides scalable, transparent alternatives to deep contextual embeddings suitable for domain-specific and end-to-end NLP applications.

Abstract

The Tsetlin Machine (TM) architecture has recently demonstrated effectiveness in Machine Learning (ML), particularly within Natural Language Processing (NLP). It has been utilized to construct word embedding using conjunctive propositional clauses, thereby significantly enhancing our understanding and interpretation of machine-derived decisions. The previous approach performed the word embedding over a sequence of input words to consolidate the information into a cohesive and unified representation. However, that approach encounters scalability challenges as the input size increases. In this study, we introduce a novel approach incorporating two-phase training to discover contextual embeddings of input sequences. Specifically, this method encapsulates the knowledge for each input word within the dataset's vocabulary, subsequently constructing embeddings for a sequence of input words utilizing the extracted knowledge. This technique not only facilitates the design of a scalable model but also preserves interpretability. Our experimental findings revealed that the proposed method yields competitive performance compared to the previous approaches, demonstrating promising results in contrast to human-generated benchmarks. Furthermore, we applied the proposed approach to sentiment analysis on the IMDB dataset, where the TM embedding and the TM classifier, along with other interpretable classifiers, offered a transparent end-to-end solution with competitive performance.

Figures (4)

• Figure 1: Illustration of Tsetlin automata in a Tsetlin Machine: Demonstrating the processing of input $X$ through multiple states to form a clause.
• Figure 2: CoTM architecture: Depicting the process flow from input $X$ through matching, updating, weighting, thresholding, and predicting output $Y$ with feedback.
• Figure 3: Proposed two-phase architecture based on TM-AE for embedding: Phase 1 involves training documents per word and for all the vocabulary, and Phase 2 focuses on clause-based embedding with word knowledge.
• Figure 4: Examples of sentence augmentation using word embeddings for sentiment analysis.