Return of the Encoder: Maximizing Parameter Efficiency for SLMs

TL;DR

The paper argues that encoder-decoder architectures retain superior efficiency for small-scale language models, outperforming decoder-only models in latency and throughput on edge devices due to one-time input processing and a clean separation of understanding and generation. It introduces a novel cross-architecture knowledge distillation framework that allows encoder-decoder models to inherit capabilities from large decoder-only teachers while preserving architectural advantages. The study extends to vision-language tasks by integrating RoPE and ViT-based vision encoders, demonstrating practical gains in multimodal reasoning at sub-billion parameter budgets. Across GPU, CPU, and NPU hardware, encoder-decoder models show consistent improvements in latency and throughput, and the proposed distillation approach yields up to several percentage-point gains on diverse benchmarks, underscoring the importance of architectural choices at small scales for edge deployment.

Abstract

The dominance of large decoder-only language models has overshadowed encoder-decoder architectures, despite their fundamental efficiency advantages in sequence processing. For small language models (SLMs) - those with 1 billion parameters or fewer - our systematic analysis across GPU, CPU, and NPU platforms reveals that encoder-decoder architectures achieve 47% lower first-token latency and 4.7x higher throughput compared to decoder-only models on edge devices. These gains may be attributed to encoder-decoder's one-time input processing and efficient separation of understanding and generation phases. We introduce a novel knowledge distillation framework that enables encoder-decoder models to leverage capabilities from large scalable decoder-only teachers while preserving their architectural advantages, achieving up to 6 average performance points improvement across diverse tasks, with significant gains in asymmetric sequence tasks where input and output distributions can benefit from different processing approaches. When combined with modern advances like Rotary Positional Embeddings (RoPE) and Vision encoders, our systematic investigation demonstrates that encoder-decoder architectures provide a more practical path toward deploying capable language models in resource-constrained environments. Our findings challenge the prevailing trend toward decoder-only scaling, showing that architectural choices become increasingly crucial as parameter budgets decrease, particularly for on-device and edge deployments where computational efficiency is paramount.

Figures (4)

• Figure 1: Architectural Efficiency in SLMs. Left: Comparison of architectures where encoder-decoder creates a fixed input representation with KV cache only for output, while decoder-only requires growing KV caches for both input and output. Top right: Inference time scaling with input length, showing encoder-decoder's efficient fixed-representation approach versus decoder-only's steeper computational growth. Bottom right: Performance across tasks showing encoder-decoder's advantages at fixed compute budget, further enhanced by KD.
• Figure 2: Performance across various model scales across top two architectures (2/3-1/3 enc-dec vs dec-only).
• Figure 3: Vision Language Encoder-Decoder Architecture.
• Figure 4: Performance comparison across vision-language tasks. Despite equal parameter constraints (800M), our encoder-decoder architecture consistently outperforms the decoder-only baseline.