Understanding QA generation: Extracting Parametric and Contextual Knowledge with CQA for Low Resource Bangla Language

TL;DR

This paper introduces BanglaCQA, the first counterfactual QA dataset for Bangla designed to disentangle parametric and contextual knowledge in QA models. It combines encoder-decoder fine-tuning (BanglaT5, mT5) and decoder-only LLM prompting with Chain-of-Thought to separately evaluate internal model knowledge versus context-derived reasoning, using both automated semantic evaluators and human judgments. Key findings show CoT prompting substantially improves parametric reasoning in counterfactual settings, while encoder-decoder models struggle with parametric CF tasks, highlighting architecture- and prompting-dependent knowledge utilization. BanglaCQA provides a framework for evaluating and guiding robust QA in low-resource languages and motivates temporally adaptive and multi-reference evaluation strategies.

Abstract

Question-Answering (QA) models for low-resource languages like Bangla face challenges due to limited annotated data and linguistic complexity. A key issue is determining whether models rely more on pre-encoded (parametric) knowledge or contextual input during answer generation, as existing Bangla QA datasets lack the structure required for such analysis. We introduce BanglaCQA, the first Counterfactual QA dataset in Bangla, by extending a Bangla dataset while integrating counterfactual passages and answerability annotations. In addition, we propose fine-tuned pipelines for encoder-decoder language-specific and multilingual baseline models, and prompting-based pipelines for decoder-only LLMs to disentangle parametric and contextual knowledge in both factual and counterfactual scenarios. Furthermore, we apply LLM-based and human evaluation techniques that measure answer quality based on semantic similarity. We also present a detailed analysis of how models perform across different QA settings in low-resource languages, and show that Chain-of-Thought (CoT) prompting reveals a uniquely effective mechanism for extracting parametric knowledge in counterfactual scenarios, particularly in decoder-only LLMs. Our work not only introduces a novel framework for analyzing knowledge sources in Bangla QA but also uncovers critical findings that open up broader directions for counterfactual reasoning in low-resource language settings.

Figures (10)

• Figure 1: Parametric vs Contextual Question Answering (QA) in Factual and Counterfactual Settings
• Figure 2: Evaluation pipeline for disentangling parametric and contextual knowledge in QA. Left: Prompt-based inference using large language models (LLMs) to generate both parametric and contextual answers. Right: Fine-tuning-based evaluation using T5 variants finetuned on BanglaCQA. Both paradigms are evaluated via automated LLM-based and human evaluations to measure answer similarity with respect to both knowledge types.
• Figure 3: Example of temporal mismatch where a model-generated answer is penalized for being more up-to-date than the reference
• Figure 4: Example showing multiple valid answers due to variations in model interpretation and reference grounding.
• Figure 5: Details of annotator instructor mail.