Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Sparse Tsetlin Machine: Sparse Representation with Active Literals

Sebastian Østby, Tobias M. Brambo, Sondre Glimsdal

TL;DR

The paper tackles the inefficiency of traditional Tsetlin Machines when handling sparse data by introducing the Sparse Tsetlin Machine (STM) and Active Literals (AL). By combining CSR-based input, per-class AL gatekeeping, and a dynamically pruned sparse memory space, STM focuses resources on discriminative literals and discards irrelevant zeros and negations. Empirical results across eight datasets show competitive accuracy with substantial improvements in memory and training efficiency, and successful deployment on large-scale text corpora demonstrates practical scalability. Overall, STM offers a principled, interpretable, and scalable approach to sparse clause representation for rule-based learning.

Abstract

This paper introduces the Sparse Tsetlin Machine (STM), a novel Tsetlin Machine (TM) that processes sparse data efficiently. Traditionally, the TM does not consider data characteristics such as sparsity, commonly seen in NLP applications and other bag-of-word-based representations. Consequently, a TM must initialize, store, and process a significant number of zero values, resulting in excessive memory usage and computational time. Previous attempts at creating a sparse TM have predominantly been unsuccessful, primarily due to their inability to identify which literals are sufficient for TM training. By introducing Active Literals (AL), the STM can focus exclusively on literals that actively contribute to the current data representation, significantly decreasing memory footprint and computational time while demonstrating competitive classification performance.

The Sparse Tsetlin Machine: Sparse Representation with Active Literals

TL;DR

The paper tackles the inefficiency of traditional Tsetlin Machines when handling sparse data by introducing the Sparse Tsetlin Machine (STM) and Active Literals (AL). By combining CSR-based input, per-class AL gatekeeping, and a dynamically pruned sparse memory space, STM focuses resources on discriminative literals and discards irrelevant zeros and negations. Empirical results across eight datasets show competitive accuracy with substantial improvements in memory and training efficiency, and successful deployment on large-scale text corpora demonstrates practical scalability. Overall, STM offers a principled, interpretable, and scalable approach to sparse clause representation for rule-based learning.

Abstract

This paper introduces the Sparse Tsetlin Machine (STM), a novel Tsetlin Machine (TM) that processes sparse data efficiently. Traditionally, the TM does not consider data characteristics such as sparsity, commonly seen in NLP applications and other bag-of-word-based representations. Consequently, a TM must initialize, store, and process a significant number of zero values, resulting in excessive memory usage and computational time. Previous attempts at creating a sparse TM have predominantly been unsuccessful, primarily due to their inability to identify which literals are sufficient for TM training. By introducing Active Literals (AL), the STM can focus exclusively on literals that actively contribute to the current data representation, significantly decreasing memory footprint and computational time while demonstrating competitive classification performance.
Paper Structure (13 sections, 8 equations, 8 figures, 2 tables)

This paper contains 13 sections, 8 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Tsetlin Machines
  3. Coalesced Tsetlin Machine
  4. Sparse Tsetlin Machine w/Active Literals
  5. Active Literals
  6. Sparse Memory Space
  7. Updating the memory space
  8. Empirical results
  9. Experiments details
  10. Datasets
  11. Results
  12. Deploying STM on large scale text corpus
  13. Conclusion

Figures (8)

  • Figure 1: Original dense input and corresponding CSR representation.
  • Figure 2: Structure of a sparse clause $C_{j}$.
  • Figure 3: Sparse TAs are defined by assigning a lower threshold $t$ for states. TAs reduced below the allowed threshold get removed from clauses.
  • Figure 4: STMs uses AL as a gatekeeper to introduce literals to clause during feedback.
  • Figure 5: STM training time in seconds (s) per epoch running 100 epochs on the SUBJ dataset with varying vocabulary size $\hat{V}$.
  • ...and 3 more figures