SUTRA: Scalable Multilingual Language Model Architecture

TL;DR

SUTRA introduces a decoupled, multilingual LLM architecture that separates concept learning from language processing, enabling scalable alignment across more than 50 languages. Built on a Transformer backbone with Mixture-of-Experts and language-specific NMT encoders/decoders, it achieves strong multilingual performance, robustness across languages, and online, up-to-date responses. The three-phase training (concept learning, language learning, language alignment) and a specialized multilingual tokenizer reduce tokenization costs while preserving linguistic nuance. Real-time evaluation shows SUTRA-Online can outperform search-augmented baselines on freshness tasks, suggesting substantial practical impact for global, inclusive AI deployment.

Abstract

In this paper, we introduce SUTRA, multilingual Large Language Model architecture capable of understanding, reasoning, and generating text in over 50 languages. SUTRA's design uniquely decouples core conceptual understanding from language-specific processing, which facilitates scalable and efficient multilingual alignment and learning. Employing a Mixture of Experts framework both in language and concept processing, SUTRA demonstrates both computational efficiency and responsiveness. Through extensive evaluations, SUTRA is demonstrated to surpass existing models like GPT-3.5, Llama2 by 20-30% on leading Massive Multitask Language Understanding (MMLU) benchmarks for multilingual tasks. SUTRA models are also online LLMs that can use knowledge from the internet to provide hallucination-free, factual and up-to-date responses while retaining their multilingual capabilities. Furthermore, we explore the broader implications of its architecture for the future of multilingual AI, highlighting its potential to democratize access to AI technology globally and to improve the equity and utility of AI in regions with predominantly non-English languages. Our findings suggest that SUTRA not only fills pivotal gaps in multilingual model capabilities but also establishes a new benchmark for operational efficiency and scalability in AI applications.

Figures (4)

• Figure 1: SUTRA is a novel multilingual large language model architecture that is trained by decoupling concept learning from language learning. The input is processed through a multilingual concept encoder, followed by the concept model and finally through a multilingual concept decoder to generate the output response.
• Figure 2: Expert Mixture Layer Configuration. Input vectors are routed to a subset of the available experts, specifically 2 out of 8, by a specialized router. The aggregate output of this layer is the sum of the individual outputs, each weighted accordingly. Each expert comprises a feedforward module similar to those found in conventional transformer models.
• Figure 3: The same concepts (umbrella, house, dog) when expressed in different languages (English, Hindi) can be mapped to quite different embedding vectors (left). In order to achieve multilingual encoders and decoders which map concepts in different languages to a common concept space, the these embedding vectors need to be aligned (middle). It can be seen that after the multilingual concept alignment stage, the same concepts (umbrella, house, dog) are now mapped to similar embedding vectors, even though they are expressed in different languages (right). Our specialized Neural Machine Translation (NMT) based encoders and decoders, can apply the same principle to master multi-language translation and ensure concept consistency across languages.
• Figure 4: Conversation Data Topic Distribution. In the following plot we are are showing topic distribution of over 1M sampled conversations. Inspection of cluster centroids reveals that this is a rich and diverse data covering wide range of topics.