Towards Robust Knowledge Representations in Multilingual LLMs for Equivalence and Inheritance based Consistent Reasoning
Gaurav Arora, Srujana Merugu, Shreya Jain, Vaibhav Saxena
TL;DR
The paper investigates whether multilingual LLMs possess language-agnostic representations for reasoning by equivalence and inheritance. It introduces parallel benchmarks across six languages to quantify cross-language conflicts and develops Compositional Representations (CoRe) to bridge distant language representations, improving consistency. Key findings show substantial cross-language conflicts (up to 57.5% for equivalence and up to ~37% for inheritance) and demonstrate that CoRe can reduce conflicts by up to 4.7% and improve some downstream tasks by ~14%. The work highlights systemic gaps in multilingual reasoning and provides a concrete, transferable method to enhance cross-language knowledge transfer in LLMs.
Abstract
Reasoning and linguistic skills form the cornerstone of human intelligence, facilitating problem-solving and decision-making. Recent advances in Large Language Models (LLMs) have led to impressive linguistic capabilities and emergent reasoning behaviors, fueling widespread adoption across application domains. However, LLMs still struggle with complex reasoning tasks, highlighting their systemic limitations. In this work, we focus on evaluating whether LLMs have the requisite representations to reason using two foundational relationships: "equivalence" and "inheritance". We introduce novel tasks and benchmarks spanning six languages and observe that current SOTA LLMs often produce conflicting answers to the same questions across languages in 17.3-57.5% of cases and violate inheritance constraints in up to 37.2% cases. To enhance consistency across languages, we propose novel "Compositional Representations" where tokens are represented as composition of equivalent tokens across languages, with resulting conflict reduction (up to -4.7%) indicating benefits of shared LLM representations.