Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Objective and Unbiased Decision Assessments with LLM-Enhanced Hierarchical Attention Networks

Junhua Liu, Kwan Hui Lim, Roy Ka-Wei Lee

TL;DR

This work proposes BGM-HAN, an enhanced Hierarchical Attention Network with Byte-Pair Encoding, Gated Residual Connections and Multi-Head Attention, and proposes a Shortlist-Analyse-Recommend agentic workflow, which simulate real-world decision-making and significantly improves on both human judgment and alternative models.

Abstract

How objective and unbiased are we while making decisions? This work investigates cognitive bias identification in high-stake decision making process by human experts, questioning its effectiveness in real-world settings, such as candidates assessments for university admission. We begin with a statistical analysis assessing correlations among different decision points among in the current process, which discovers discrepancies that imply cognitive bias and inconsistency in decisions. This motivates our exploration of bias-aware AI-augmented workflow that surpass human judgment. We propose BGM-HAN, an enhanced Hierarchical Attention Network with Byte-Pair Encoding, Gated Residual Connections and Multi-Head Attention. Using it as a backbone model, we further propose a Shortlist-Analyse-Recommend (SAR) agentic workflow, which simulate real-world decision-making. In our experiments, both the proposed model and the agentic workflow significantly improves on both human judgment and alternative models, validated with real-world data.

Towards Objective and Unbiased Decision Assessments with LLM-Enhanced Hierarchical Attention Networks

TL;DR

This work proposes BGM-HAN, an enhanced Hierarchical Attention Network with Byte-Pair Encoding, Gated Residual Connections and Multi-Head Attention, and proposes a Shortlist-Analyse-Recommend agentic workflow, which simulate real-world decision-making and significantly improves on both human judgment and alternative models.

Abstract

How objective and unbiased are we while making decisions? This work investigates cognitive bias identification in high-stake decision making process by human experts, questioning its effectiveness in real-world settings, such as candidates assessments for university admission. We begin with a statistical analysis assessing correlations among different decision points among in the current process, which discovers discrepancies that imply cognitive bias and inconsistency in decisions. This motivates our exploration of bias-aware AI-augmented workflow that surpass human judgment. We propose BGM-HAN, an enhanced Hierarchical Attention Network with Byte-Pair Encoding, Gated Residual Connections and Multi-Head Attention. Using it as a backbone model, we further propose a Shortlist-Analyse-Recommend (SAR) agentic workflow, which simulate real-world decision-making. In our experiments, both the proposed model and the agentic workflow significantly improves on both human judgment and alternative models, validated with real-world data.

Paper Structure

This paper contains 51 sections, 17 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Data Analysis
  4. Dataset
  5. Decisions Correlation Analysis
  6. Methodology
  7. BGM-HAN
  8. Base Architecture
  9. Byte-Pair Encoding and Embedding
  10. Multi-Head Attention
  11. Gated Residual Network
  12. Agentic Workflow
  13. Decision Agents
  14. Shortlisting Agent ($\mathcal{A}_S$)
  15. Analysis Agent ($\mathcal{A}_A$)
  16. ...and 36 more sections

Figures (3)

  • Figure 1: University Admission Decision Process: overview of current workflow and possible agentic augmentation
  • Figure 2: Architecture overview of the proposed BGM-HAN model. The multi-level model learns features from token to sentence to field. At each level, the data will go through layer normalisation, multi-head self-attention (MHA), gated residual connection (GRC), mean pooling to form the higher level embeddings. The embeddings are then concatenated and reshaped into 3D tensors to continue with the next level processing.
  • Figure 3: Correlation matrix of different decision points in the admission assessments process.