Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Consistent Joint Decision-Making with Heterogeneous Learning Models

Hossein Rajaby Faghihi, Parisa Kordjamshidi

TL;DR

The paper tackles the problem of achieving consistent interdependent decisions from heterogeneous models without retraining. It introduces an ILP-based framework that maps raw model outputs to globally comparable weights using priors, entropy-based confidence, and expected accuracy, and then maximizes a global objective under task constraints. By integrating these factors within the DomiKnowS pipeline, the approach yields improved constraint satisfaction and set correctness across procedural reasoning and hierarchical classification tasks. The findings demonstrate that globally optimized, uncertainty-aware scoring enhances practical utility of multi-model predictions, though replication may depend on optimization tooling and problem characteristics. This work advances decision fusion for complex, interrelated outputs in AI systems.

Abstract

This paper introduces a novel decision-making framework that promotes consistency among decisions made by diverse models while utilizing external knowledge. Leveraging the Integer Linear Programming (ILP) framework, we map predictions from various models into globally normalized and comparable values by incorporating information about decisions' prior probability, confidence (uncertainty), and the models' expected accuracy. Our empirical study demonstrates the superiority of our approach over conventional baselines on multiple datasets.

Consistent Joint Decision-Making with Heterogeneous Learning Models

TL;DR

The paper tackles the problem of achieving consistent interdependent decisions from heterogeneous models without retraining. It introduces an ILP-based framework that maps raw model outputs to globally comparable weights using priors, entropy-based confidence, and expected accuracy, and then maximizes a global objective under task constraints. By integrating these factors within the DomiKnowS pipeline, the approach yields improved constraint satisfaction and set correctness across procedural reasoning and hierarchical classification tasks. The findings demonstrate that globally optimized, uncertainty-aware scoring enhances practical utility of multi-model predictions, though replication may depend on optimization tooling and problem characteristics. This work advances decision fusion for complex, interrelated outputs in AI systems.

Abstract

This paper introduces a novel decision-making framework that promotes consistency among decisions made by diverse models while utilizing external knowledge. Leveraging the Integer Linear Programming (ILP) framework, we map predictions from various models into globally normalized and comparable values by incorporating information about decisions' prior probability, confidence (uncertainty), and the models' expected accuracy. Our empirical study demonstrates the superiority of our approach over conventional baselines on multiple datasets.
Paper Structure (28 sections, 3 equations, 2 figures, 5 tables)

This paper contains 28 sections, 3 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Prior Probability (Output Size)
  4. Entropy and Confidence
  5. Expected Models' Accuracy
  6. Empirical Study
  7. Metrics and Evaluation
  8. Tasks
  9. Procedural Reasoning
  10. Hierarchical Classification
  11. Results
  12. Conclusion
  13. Datasets & Baselines
  14. Animal/Flower/Food
  15. 20News
  16. ...and 13 more sections

Figures (2)

  • Figure 1: An overview of the proposed solution to maintain consistency between model decisions during inference via ILP optimization. The task used as an example here is the Hierarchical Image Classification task with two levels. The Green blocks represent additional components that have been added to the pipeline in this paper to guarantee the global comparability of model-generated probabilities.
  • Figure 2: An example from the Propara dataset taken from faghihi2023role. '-' refers to the entity not existing; '?' refers to the entity whose location is unclear.