Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NADIR: Differential Attention Flow for Non-Autoregressive Transliteration in Indic Languages

Lakshya Tomar, Vinayak Abrol, Puneet Agarwal

TL;DR

NADIR addresses the latency-accuracy trade-off in transliteration by marrying a Differential Transformer with a Mixture-of-Experts in a non-autoregressive architecture, tailored for multilingual Indic scripts. It uses implicit termination and a composite loss to avoid explicit length predictors, enabling fully parallel decoding. Experiments on the Aksharantar dataset show NADIR achieves over a 13x speed-up with CER and WAcc close to AR baselines, and ablations reveal that differential attention and MoE jointly reduce NAR hallucinations (insertions, substitutions, omissions, repetitions). The approach provides a practical blueprint for deploying fast, reliable NAR systems in real-time, large-scale multilingual settings and can be extended to other structured generation tasks.

Abstract

In this work, we argue that not all sequence-to-sequence tasks require the strong inductive biases of autoregressive (AR) models. Tasks like multilingual transliteration, code refactoring, grammatical correction or text normalization often rely on local dependencies where the full modeling capacity of AR models can be overkill, creating a trade-off between their high accuracy and high inference latency. While non-autoregressive (NAR) models offer speed, they typically suffer from hallucinations and poor length control. To explore this trade-off, we focus on the multilingual transliteration task in Indic languages and introduce NADIR, a novel NAR architecture designed to strike a balance between speed and accuracy. NADIR integrates a Differential Transformer and a Mixture-of-Experts mechanism, enabling it to robustly model complex character mappings without sequential dependencies. NADIR achieves over a 13x speed-up compared to the state-of-the-art AR baseline. It maintains a competitive mean Character Error Rate of 15.78%, compared to 14.44% for the AR model and 21.88% for a standard NAR equivalent. Importantly, NADIR reduces Repetition errors by 49.53%, Substitution errors by 24.45%, Omission errors by 32.92%, and Insertion errors by 16.87%. This work provides a practical blueprint for building fast and reliable NAR systems, effectively bridging the gap between AR accuracy and the demands of real-time, large-scale deployment.

NADIR: Differential Attention Flow for Non-Autoregressive Transliteration in Indic Languages

TL;DR

NADIR addresses the latency-accuracy trade-off in transliteration by marrying a Differential Transformer with a Mixture-of-Experts in a non-autoregressive architecture, tailored for multilingual Indic scripts. It uses implicit termination and a composite loss to avoid explicit length predictors, enabling fully parallel decoding. Experiments on the Aksharantar dataset show NADIR achieves over a 13x speed-up with CER and WAcc close to AR baselines, and ablations reveal that differential attention and MoE jointly reduce NAR hallucinations (insertions, substitutions, omissions, repetitions). The approach provides a practical blueprint for deploying fast, reliable NAR systems in real-time, large-scale multilingual settings and can be extended to other structured generation tasks.

Abstract

In this work, we argue that not all sequence-to-sequence tasks require the strong inductive biases of autoregressive (AR) models. Tasks like multilingual transliteration, code refactoring, grammatical correction or text normalization often rely on local dependencies where the full modeling capacity of AR models can be overkill, creating a trade-off between their high accuracy and high inference latency. While non-autoregressive (NAR) models offer speed, they typically suffer from hallucinations and poor length control. To explore this trade-off, we focus on the multilingual transliteration task in Indic languages and introduce NADIR, a novel NAR architecture designed to strike a balance between speed and accuracy. NADIR integrates a Differential Transformer and a Mixture-of-Experts mechanism, enabling it to robustly model complex character mappings without sequential dependencies. NADIR achieves over a 13x speed-up compared to the state-of-the-art AR baseline. It maintains a competitive mean Character Error Rate of 15.78%, compared to 14.44% for the AR model and 21.88% for a standard NAR equivalent. Importantly, NADIR reduces Repetition errors by 49.53%, Substitution errors by 24.45%, Omission errors by 32.92%, and Insertion errors by 16.87%. This work provides a practical blueprint for building fast and reliable NAR systems, effectively bridging the gap between AR accuracy and the demands of real-time, large-scale deployment.
Paper Structure (20 sections, 6 equations, 2 figures, 6 tables)

This paper contains 20 sections, 6 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. NADIR (Non-Autoregressive Differential Intelligent Router)
  4. Multi-head Differential Attention:
  5. Mixture-of-Expert Module:
  6. Implicit Sequence Termination and Training Objective
  7. Why Differential Transformer Helps in Transliteration
  8. Experimental Setup
  9. Dataset
  10. Model Training Setup
  11. Experimental Results
  12. Main Result
  13. Indic$\rightarrow$Roman direction:
  14. Roman$\rightarrow$Indic direction
  15. Ablation Study
  16. ...and 5 more sections

Figures (2)

  • Figure 1: NADIR: Architecture Overview showing pre-processing, multiple encoder and non-autoregressive decoder blocks. On the right, we show an expanded view of the encoder block that is built upon differential transformer layers combined with MoE-based routing, enabling efficient and expressive context modelling.
  • Figure 2: Inference Time vs. Batch Size (log scale). NADIR achieves significantly lower inference latency compared to the AR-based IndicXLIT, especially at large batch sizes.