Two Causal Principles for Improving Visual Dialog

TL;DR

The paper reframes Visual Dialog as a causal inference problem, identifying two principles: (P1) remove direct history-to-answer shortcuts and (P2) mitigate an unobserved confounder that induces spurious history-question-answer correlations. Using do-calculus, the authors derive an interventional objective and propose three practical approximations—Question Type weighting, Answer Score Sampling, and Hidden Dictionary Learning—to implement P2. Across multiple baseline VisDial models and extensive experiments on VisDial v1.0, these principles yield substantial NDCG gains and even lead to state-of-the-art leaderboard results with simple ensembles. The work demonstrates that explicit causal reasoning can dramatically improve dialog systems, by eliminating bias and better aligning with human preferences in ranking answers. It also provides model-agnostic techniques that can be adopted by a wide range of VisDial architectures.

Abstract

This paper unravels the design tricks adopted by us, the champion team MReaL-BDAI, for Visual Dialog Challenge 2019: two causal principles for improving Visual Dialog (VisDial). By "improving", we mean that they can promote almost every existing VisDial model to the state-of-the-art performance on the leader-board. Such a major improvement is only due to our careful inspection on the causality behind the model and data, finding that the community has overlooked two causalities in VisDial. Intuitively, Principle 1 suggests: we should remove the direct input of the dialog history to the answer model, otherwise a harmful shortcut bias will be introduced; Principle 2 says: there is an unobserved confounder for history, question, and answer, leading to spurious correlations from training data. In particular, to remove the confounder suggested in Principle 2, we propose several causal intervention algorithms, which make the training fundamentally different from the traditional likelihood estimation. Note that the two principles are model-agnostic, so they are applicable in any VisDial model. The code is available at https://github.com/simpleshinobu/visdial-principles.

Figures (5)

• Figure 1: Causal graphs of VisDial models (baseline and ours). $H$: dialog history. $I$: image. $Q$: question. $V$: visual knowledge. $A$: answer. $U$: user preference. Shaded $U$ denotes unobserved confounder. See Section \ref{['subsection:3_2']} for detailed definitions.
• Figure 2: The illustrative motivations of the two causal principles: (a) P1 and (b) P2.
• Figure 3: Example of confounder, do-operator and sketch causal graphs of our three attempts to de-confounder
• Figure 4: Qualitative results of the baseline and baseline with P1 on the validation set of VisDial v1.0. The numbers in brackets in ranked $A$ denote relevance scores. Red boxes denote that the selected candidates of the baseline model influenced by the shortcut (e.g., word matching) from the dialog history. For the baseline with P1, it does not make such biased shortcut choices. More details can be found in Section \ref{['subsection:6_5']}.
• Figure 5: Qualitative examples of the ranked candidates of baseline and baseline with P2. We also give some key rank changes for boosting NDCG performance by implementing P2. These examples are taken from the validation set of VisDial v1.0.