Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MKA: Leveraging Cross-Lingual Consensus for Model Abstention

Sharad Duwal

TL;DR

The paper tackles reliability and hallucination in LLMs by introducing MKA, a pipeline that calibrates model confidence through cross-lingual consensus. It translates questions and options into auxiliary languages, prompts the model across these languages, and aggregates answers via centroid polling, using a cosine-similarity-based confidence measure to decide abstention. Evaluated on multilingual MCQA benchmarks (MMLU) across six target languages and three auxiliary sets, the approach yields substantial gains for some low- and mid-resource languages (e.g., Bengali with $+71.2\%$) and mixed results for others, depending on translation quality and auxiliary language choice (e.g., Yoruba, Japanese can drop). The findings support cross-lingual confidence calibration as a viable tool to reduce hallucinations, while also noting limitations due to translation artifacts and the need for improved internal multilingual reasoning in future work, with practical implications for safer and more reliable LLM deployment. $0.85$, $0.81$, and $1.5$ are among the critical thresholds and weights used in the evaluation and calibration process, respectively.

Abstract

Reliability of LLMs is questionable even as they get better at more tasks. A wider adoption of LLMs is contingent on whether they are usably factual. And if they are not, on whether they can properly calibrate their confidence in their responses. This work focuses on utilizing the multilingual knowledge of an LLM to inform its decision to abstain or answer when prompted. We develop a multilingual pipeline to calibrate the model's confidence and let it abstain when uncertain. We run several multilingual models through the pipeline to profile them across different languages. We find that the performance of the pipeline varies by model and language, but that in general they benefit from it. This is evidenced by the accuracy improvement of $71.2\%$ for Bengali over a baseline performance without the pipeline. Even a high-resource language like English sees a $15.5\%$ improvement. These results hint at possible further improvements.

MKA: Leveraging Cross-Lingual Consensus for Model Abstention

TL;DR

The paper tackles reliability and hallucination in LLMs by introducing MKA, a pipeline that calibrates model confidence through cross-lingual consensus. It translates questions and options into auxiliary languages, prompts the model across these languages, and aggregates answers via centroid polling, using a cosine-similarity-based confidence measure to decide abstention. Evaluated on multilingual MCQA benchmarks (MMLU) across six target languages and three auxiliary sets, the approach yields substantial gains for some low- and mid-resource languages (e.g., Bengali with ) and mixed results for others, depending on translation quality and auxiliary language choice (e.g., Yoruba, Japanese can drop). The findings support cross-lingual confidence calibration as a viable tool to reduce hallucinations, while also noting limitations due to translation artifacts and the need for improved internal multilingual reasoning in future work, with practical implications for safer and more reliable LLM deployment. , , and are among the critical thresholds and weights used in the evaluation and calibration process, respectively.

Abstract

Reliability of LLMs is questionable even as they get better at more tasks. A wider adoption of LLMs is contingent on whether they are usably factual. And if they are not, on whether they can properly calibrate their confidence in their responses. This work focuses on utilizing the multilingual knowledge of an LLM to inform its decision to abstain or answer when prompted. We develop a multilingual pipeline to calibrate the model's confidence and let it abstain when uncertain. We run several multilingual models through the pipeline to profile them across different languages. We find that the performance of the pipeline varies by model and language, but that in general they benefit from it. This is evidenced by the accuracy improvement of for Bengali over a baseline performance without the pipeline. Even a high-resource language like English sees a improvement. These results hint at possible further improvements.

Paper Structure

This paper contains 14 sections, 4 equations, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. MKA Pipeline
  4. Baseline
  5. Automatic Evaluation
  6. Experiments
  7. Setup
  8. Metrics
  9. Results
  10. Conclusion
  11. Limitations and Future Work
  12. Gemma Models
  13. MKA Pipeline Output Example
  14. Confidence Cutoff Analyses

Figures (10)

  • Figure 1: MKA Pipeline confusion matrix
  • Figure 2: Effective Accuracy versus Confidence Cutoff
  • Figure 3: Answered Accuracy versus Coverage
  • Figure 5: Effective Accuracy versus Confidence Cutoff for the MKA pipeline on the Gemma models.
  • Figure 6: Answered Accuracy versus Confidence Cutoff.
  • ...and 5 more figures