EmbBERT-Q: Breaking Memory Barriers in Embedded NLP

TL;DR

This work tackles the challenge of running NLP on ultra-constrained devices by introducing EmbBERT-Q, a tiny language model designed for microcontrollers under a strict memory budget. It couples a Nano Embedder, Efficient Attention, weighted-difference skip connections, and 1D convolutions to dramatically reduce activations while preserving task performance, achieving a total footprint of about 781 kB after 8-bit quantization. Evaluations on TinyNLP and GLUE show EmbBERT-Q delivers competitive accuracy with far lower memory than prior SOTA tiny models, and it maintains robustness under quantization with minimal performance loss. The work provides extensive ablations, demonstrates reproducibility with open-source code, and outlines practical directions (e.g., LoRA, distillation) for further compression, underscoring the feasibility of on-device natural language understanding on tiny hardware.

Abstract

Large Language Models (LLMs) have revolutionized natural language processing, setting new standards across a wide range of applications. However, their relevant memory and computational demands make them impractical for deployment on technologically-constrained tiny devices such as wearable devices and Internet-of-Things units. To address this limitation, we introduce EmbBERT-Q, a novel tiny language model specifically designed for tiny devices with stringent memory constraints. EmbBERT-Q achieves state-of-the-art (SotA) accuracy in Natural Language Processing tasks in this scenario, with a total memory footprint (weights and activations) of just 781 kB, representing a 25x reduction in size with respect to SotA models. By combining architectural innovations with hardware-compatible 8-bit quantization, EmbBERT-Q consistently outperforms several baseline models scaled down to a 2 MB memory budget (i.e., the maximum memory typically available in tiny devices), including heavily compressed versions of BERT and MAMBA. Extensive experimental evaluations on both a selected benchmark dataset, TinyNLP, specifically curated to evaluate Tiny Language Models in NLP tasks and real-world scenarios, and the GLUE benchmark, demonstrate EmbBERT-Q ability to deliver competitive accuracy with respect to existing approaches, achieving an unmatched balance between memory and performance. To ensure the complete and immediate reproducibility of all our results, we release all code, scripts, and model checkpoints at https://github.com/RiccardoBravin/tiny-LLM.

Figures (6)

• Figure 1: The EmbBert architecture, our purpose-built Language Model for Tiny Devices and extreme resource constraints.It is composed of a Nano Embedder and a series of computational blocks in which our main contribution is the use of the new structure which leverages the original skip connection for further processing through the use of a combination of convolutional and fully connected layers plus the use of a weighted difference with learned weights instead of the standard sum.
• Figure 2: Model's average accuracy on the TinyNLP benchmarkof the models in the ablation study.
• Figure 3: Model's scores on the GLUE benchmark of the models in the ablation study.
• Figure 5: Memory layout of the Standard Embedder (left) and Nano Embedder (right) layer. $\ell$ is the sentence length, $d$ is the embedding dimension, the trapezoid with fc is used to represent fully connected layers. The dashed gray box shows the operations which require the maximum acivations. The Nano Embedder manages to decrease the number of weights while still keeping about the same size for the activations.
• Figure 6: Memory layout of the Attention (left) and Efficient Attention (right) layers. $\ell$ is the sentence length, $d$ is the embedding dimension, the trapezoid with fc is used to represent fully connected layers. The dashed gray box shows the operations which require the maximum acivations. As can be see just from the width of the two modules, Efficient Attention uses a quite lower activation size.