Parameter-Efficient Transformer Embeddings

TL;DR

The paper addresses the large parameter footprint of traditional token embeddings in transformers by introducing a deterministic, Fourier-based embedding mapped from normalized token IDs, refined by a small MLP with a residual connection. This PETE approach replaces the $V \times d$ embedding table with a fixed Fourier representation plus a compact learnable network, achieving competitive performance on SNLI/MNLI and STS-B with substantially fewer parameters and faster training. Empirical results show that, at modest model capacity, the method matches or slightly surpasses learned embeddings, and further parameter reductions via reduced MLP dimensions yield meaningful gains. The work suggests substantial potential for scalable, memory-efficient language models and encourages large-scale validation and exploration of alternative bases.

Abstract

Embedding layers in transformer-based NLP models typically account for the largest share of model parameters, scaling with vocabulary size but not yielding performance gains proportional to scale. We propose an alternative approach in which token embedding vectors are first generated deterministically, directly from the token IDs using a Fourier expansion of their normalized values, followed by a lightweight multilayer perceptron (MLP) that captures higher-order interactions. We train standard transformers and our architecture on natural language inference tasks (SNLI and MNLI), and evaluate zero-shot performance on sentence textual similarity (STS-B). Our results demonstrate that the proposed method achieves competitive performance using significantly fewer parameters, trains faster, and operates effectively without the need for dropout. This proof-of-concept study highlights the potential for scalable, memory-efficient language models and motivates further large-scale experimentation based on our findings.