HITgram: A Platform for Experimenting with n-gram Language Models

TL;DR

HITgram addresses the inaccessibility of large language models by delivering a lightweight platform for n-gram experimentation that supports unigrams through 4-grams, with smoothing, context-sensitive weighting, and dynamic corpus management. Implemented in Java with a GUI, it enables efficient tokenization, corpus management, and rapid n-gram construction on resource-constrained hardware. Empirical results show HITgram achieving high throughput (≈50k tokens/s) and fast generation times for 2- and 4-grams on medium-to-large corpora, while using perplexity to assess predictive performance. The work underscores HITgram's practical value for education, preprocessing, autocomplete, and domains requiring interpretable, efficient NLP tools, and outlines a roadmap for multilingual support, parallelization, and richer evaluation metrics.

Abstract

Large language models (LLMs) are powerful but resource intensive, limiting accessibility. HITgram addresses this gap by offering a lightweight platform for n-gram model experimentation, ideal for resource-constrained environments. It supports unigrams to 4-grams and incorporates features like context sensitive weighting, Laplace smoothing, and dynamic corpus management to e-hance prediction accuracy, even for unseen word sequences. Experiments demonstrate HITgram's efficiency, achieving 50,000 tokens/second and generating 2-grams from a 320MB corpus in 62 seconds. HITgram scales efficiently, constructing 4-grams from a 1GB file in under 298 seconds on an 8 GB RAM system. Planned enhancements include multilingual support, advanced smoothing, parallel processing, and model saving, further broadening its utility.

Figures (2)

• Figure 1: Plot of the time taken by HITgram: Time taken to load and build $n$-gram models, by varying corpus size and $n$.
• Figure 2: HITgram in Action: A text corpus generated from a user-uploaded PDF, followed by downstream processing.