Do Language Models Reason Across Languages?

TL;DR

This work introduces a controlled multilingual two-hop QA task that requires integrating information from two documents in different languages to answer a question. It reveals that language models often fail to decompose reasoning faithfully across languages, particularly when the second-hop document is in a different language, with unfaithfulness up to $33\%$ and compositional failure up to $18\%$. Through step-level analysis (SubQ) and prompting, the authors show that explicit sub-question decomposition can substantially mitigate compositional failures, boosting accuracy from $10.1\%$ to $66.56\%$ in multilingual settings. The study also demonstrates that reasoning is not purely step-by-step but is influenced by bridging content, distractors, and document order, indicating room for improvement via targeted prompting and training strategies. These findings highlight the potential and limitations of current multilingual LLMs and offer a practical prompting approach (SUBQ) to enhance cross-language multi-hop reasoning.

Abstract

The real-world information sources are inherently multilingual, which naturally raises a question about whether language models can synthesize information across languages. In this paper, we introduce a simple two-hop question answering setting, where answering a question requires making inferences over two multilingual documents. We find that language models are more sensitive to language variation in answer-span documents than in those providing bridging information, despite the equal importance of both documents for answering a question. Under a step-by-step sub-question evaluation, we further show that in up to 33% of multilingual cases, models fail to infer the bridging information in the first step yet still answer the overall question correctly. This indicates that reasoning in language models, especially in multilingual settings, does not follow a faithful step-by-step decomposition. Subsequently, we show that the absence of reasoning decomposition leads to around 18% composition failure, where both sub-questions are answered correctly but fail for the final two-hop questions. To mitigate this, we propose a simple three-stage SUBQ prompting method to guide the multi-step reasoning with sub-questions, which boosts accuracy from 10.1% to 66.5%.

Figures (12)

• Figure 1: Example of multilingual two-hop QA. In Section \ref{['sec:2']}, we evaluate multilingual two-step reasoning performance with the two-hop question and the corresponding Hop-1 and Hop-2 document. In Section \ref{['sec:3']}, we conduct a sub-question evaluation to disentangle the two-step reasoning mechanism: SubQ1 infers the bridge entity and SubQ2 links the bridge entity to the final answer.
• Figure 2: Context attribution scores for faithful and unfaithful cases. The two-hop query is in English, and Lang1-Lang2 (e.g., EN-ZH) indicates the languages of Hop-1 and Hop-2 documents.
• Figure 3: The impact of inserting relevant and irrelevant distractors between Hop-1 and Hop-2 documents. A distance of $d$ corresponds to $(|d|-1)$ distractors between the two hops. Positive $d$ means Hop-1 precedes Hop-2, while a negative sign means the reverse. We report the average $F1$ token scores for every unfaithful multilingual case for each query language.
• Figure 4: Three-step of SubQ Prompting. The first and second step prompt templates are in Appendix \ref{['app:zero-shot']} Fig. \ref{['fig:step-by-step-cot']}.
• Figure 5: F1 token accuracy for multilingual two-hop QA for different prompting strategies, evaluated on composition failure cases and the full dataset. Red dashed lines above the SubQ Prompt bars show the performance when using ground-truth answers for sub-questions.