Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DoubleTake: Contrastive Reasoning for Faithful Decision-Making in Medical Imaging

Daivik Patel, Shrenik Patel

TL;DR

The paper tackles discriminative decision making in medical imaging by shifting retrieval from similarity to contrastive, document-aware evidence construction and by introducing Counterfactual-Contrastive Inference (CCI) with abstention and pair-wise adjudication. It builds a triad of references from ROCO, selecting an anchor, a hard negative, and a boundary probe to probe decision boundaries. These references feed a confidence-weighted, margin-based aggregation that can abstain, with a pair-level adjudicator to handle confusion-pair scenarios. On MediConfusion, the approach achieves state-of-the-art set-level accuracy and substantially reduces confusion while maintaining faithful abstention. The work contributes reproducible reference selection protocols and demonstrates that inference-time contrastive reasoning is crucial for reliable medical AI under visual ambiguity.

Abstract

Accurate decision making in medical imaging requires reasoning over subtle visual differences between confusable conditions, yet most existing approaches rely on nearest neighbor retrieval that returns redundant evidence and reinforces a single hypothesis. We introduce a contrastive, document-aware reference selection framework that constructs compact evidence sets optimized for discrimination rather than similarity by explicitly balancing visual relevance, embedding diversity, and source-level provenance using ROCO embeddings and metadata. While ROCO provides large-scale image-caption pairs, it does not specify how references should be selected for contrastive reasoning, and naive retrieval frequently yields near-duplicate figures from the same document. To address this gap, we release a reproducible reference selection protocol and curated reference bank that enable a systematic study of contrastive retrieval in medical image reasoning. Building on these contrastive evidence sets, we propose Counterfactual-Contrastive Inference, a confidence-aware reasoning framework that performs structured pairwise visual comparisons and aggregates evidence using margin-based decision rules with faithful abstention. On the MediConfusion benchmark, our approach achieves state-of-the-art performance, improving set-level accuracy by nearly 15% relative to prior methods while reducing confusion and improving individual accuracy.

DoubleTake: Contrastive Reasoning for Faithful Decision-Making in Medical Imaging

TL;DR

The paper tackles discriminative decision making in medical imaging by shifting retrieval from similarity to contrastive, document-aware evidence construction and by introducing Counterfactual-Contrastive Inference (CCI) with abstention and pair-wise adjudication. It builds a triad of references from ROCO, selecting an anchor, a hard negative, and a boundary probe to probe decision boundaries. These references feed a confidence-weighted, margin-based aggregation that can abstain, with a pair-level adjudicator to handle confusion-pair scenarios. On MediConfusion, the approach achieves state-of-the-art set-level accuracy and substantially reduces confusion while maintaining faithful abstention. The work contributes reproducible reference selection protocols and demonstrates that inference-time contrastive reasoning is crucial for reliable medical AI under visual ambiguity.

Abstract

Accurate decision making in medical imaging requires reasoning over subtle visual differences between confusable conditions, yet most existing approaches rely on nearest neighbor retrieval that returns redundant evidence and reinforces a single hypothesis. We introduce a contrastive, document-aware reference selection framework that constructs compact evidence sets optimized for discrimination rather than similarity by explicitly balancing visual relevance, embedding diversity, and source-level provenance using ROCO embeddings and metadata. While ROCO provides large-scale image-caption pairs, it does not specify how references should be selected for contrastive reasoning, and naive retrieval frequently yields near-duplicate figures from the same document. To address this gap, we release a reproducible reference selection protocol and curated reference bank that enable a systematic study of contrastive retrieval in medical image reasoning. Building on these contrastive evidence sets, we propose Counterfactual-Contrastive Inference, a confidence-aware reasoning framework that performs structured pairwise visual comparisons and aggregates evidence using margin-based decision rules with faithful abstention. On the MediConfusion benchmark, our approach achieves state-of-the-art performance, improving set-level accuracy by nearly 15% relative to prior methods while reducing confusion and improving individual accuracy.
Paper Structure (27 sections, 16 equations, 1 figure, 5 tables)

This paper contains 27 sections, 16 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Medical Visual Question Answering Benchmarks
  4. Retrieval for Medical Image Reasoning
  5. Contrastive Reasoning at Inference Time
  6. Reliability via Abstention + Evidence Aggregation
  7. Method
  8. System Overview
  9. Contrastive, Document-Aware Reference Selection
  10. Counterfactual-Contrastive Inference
  11. Experiments
  12. Dataset and Task: MediConfusion
  13. Evaluation Metrics
  14. Baselines
  15. Implementation Details
  16. ...and 12 more sections

Figures (1)

  • Figure 1: Overview of the proposed framework. Given a query image, a contrastive triad of reference images is retrieved from an external medical image bank. Structured pairwise comparisons are aggregated by Counterfactual-Contrastive Inference (CCI) using confidence-aware voting, margin-based decisions, and adjudication for ambiguous cases.