Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reviewing the Reviewer: Graph-Enhanced LLMs for E-commerce Appeal Adjudication

Yuchen Du, Ashley Li, Zixi Huang

Abstract

Hierarchical review workflows, where a second-tier reviewer (Checker) corrects first-tier (Maker) decisions, generate valuable correction signals that encode why initial judgments failed. However, learning from these signals is hindered by information asymmetry: corrections often depend on verification actions unavailable to Makers or automated systems. We address this challenge by introducing explicit action modeling as an inferential constraint that grounds reasoning in verifiable operations rather than unconstrained text generation. We propose the Evidence-Action-Factor-Decision (EAFD) schema, a minimal representation for adjudication reasoning that prevents hallucination through operational grounding and enables learning from correction signals via explicit conflict modeling. Building on this schema, we develop a conflict-aware graph reasoning framework that: (1) constructs EAFD graphs from historical cases capturing Maker-Checker disagreements, (2) aggregates them into a retrievable knowledge base, and (3) performs top-down deductive reasoning for new cases by projecting validated resolution paths from precedents. A distinctive capability is the Request More Information (RMI) outcome: when evidence is insufficient, the system identifies precisely which verification actions remain unexecuted and generates targeted information requests. We evaluate the framework in large-scale e-commerce seller appeal adjudication. While a standard LLM-only baseline achieves only 70.8% alignment with human experts, incorporating action modeling with RMI improves alignment to 87.5%. Augmenting this with the retrieval-based knowledge graph yields the best offline performance of 95.8%. Following online deployment, the framework maintains robust performance, achieving a 96.3% alignment rate in production, demonstrating its real-world effectiveness.

Reviewing the Reviewer: Graph-Enhanced LLMs for E-commerce Appeal Adjudication

Abstract

Hierarchical review workflows, where a second-tier reviewer (Checker) corrects first-tier (Maker) decisions, generate valuable correction signals that encode why initial judgments failed. However, learning from these signals is hindered by information asymmetry: corrections often depend on verification actions unavailable to Makers or automated systems. We address this challenge by introducing explicit action modeling as an inferential constraint that grounds reasoning in verifiable operations rather than unconstrained text generation. We propose the Evidence-Action-Factor-Decision (EAFD) schema, a minimal representation for adjudication reasoning that prevents hallucination through operational grounding and enables learning from correction signals via explicit conflict modeling. Building on this schema, we develop a conflict-aware graph reasoning framework that: (1) constructs EAFD graphs from historical cases capturing Maker-Checker disagreements, (2) aggregates them into a retrievable knowledge base, and (3) performs top-down deductive reasoning for new cases by projecting validated resolution paths from precedents. A distinctive capability is the Request More Information (RMI) outcome: when evidence is insufficient, the system identifies precisely which verification actions remain unexecuted and generates targeted information requests. We evaluate the framework in large-scale e-commerce seller appeal adjudication. While a standard LLM-only baseline achieves only 70.8% alignment with human experts, incorporating action modeling with RMI improves alignment to 87.5%. Augmenting this with the retrieval-based knowledge graph yields the best offline performance of 95.8%. Following online deployment, the framework maintains robust performance, achieving a 96.3% alignment rate in production, demonstrating its real-world effectiveness.
Paper Structure (49 sections, 6 equations, 4 figures, 4 tables)

This paper contains 49 sections, 6 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Case-Based Reasoning.
  4. Structured Reasoning.
  5. Learning from Human Review Corrections.
  6. Problem Formulation
  7. Methodology
  8. EAFD Graph Schema
  9. Node Types (structure).
  10. Relation Types (structure).
  11. Action grounding (semantics and constraint).
  12. Factor abstraction (interpretation layer).
  13. Offline Knowledge Base Construction
  14. Hierarchical Graph Extraction.
  15. Action Refinement and Merging.
  16. ...and 34 more sections

Figures (4)

  • Figure 1: System overview. Offline: Extract EAFD graphs from historical cases and build a knowledge base. Online: Construct Maker graph, retrieve precedents, derive Checker graph via FAE deduction, and output adjudication decision.
  • Figure 2: The EAFD schema with four-layer hierarchy. Left: Maker's initial review leads to rejection. Right: Checker's corrective review with additional verification overturns the decision. The conflict edge (red) captures Maker-Checker disagreement as a learning signal.
  • Figure 3: Online reasoning pipeline. The system constructs the Maker graph, retrieves similar cases, aligns factors to historical paths, and applies FAE deduction ($f_q^{(C)} \to a_q^{(C)} \to e_q^{(C)}$). Final decision $\hat{d}_q^{(C)}$ is based on action status; RMI is triggered when critical evidence is missing.
  • Figure 4: Cumulative alignment rate in production (Jan 17 -- Feb 5).