HARMON-E: Hierarchical Agentic Reasoning for Multimodal Oncology Notes to Extract Structured Data

TL;DR

This work tackles the challenge of converting heterogeneous EHR oncology notes into comprehensive, structured data by introducing HARMON-E, a hierarchical agentic reasoning framework that orchestrates context-aware retrieval, multi-step LLM synthesis, and deterministic collators for patient-level data abstraction. The approach is validated on a large melanoma corpus, achieving high F1 across 103 oncology-specific attributes and delivering a 94.1% direct manual-approval rate in a real-world data curation workflow. Key contributions include a modular architecture with four pipeline styles, a formal problem formulation for multi-source data fusion, and an evaluation framework that emphasizes cross-document consistency and temporal reasoning beyond traditional NER. The results demonstrate scalable, high-fidelity extraction that can accelerate real-world evidence generation and streamline clinical research, with implications for trial matching, pharmacovigilance, and epidemiology at scale.

Abstract

Unstructured notes within the electronic health record (EHR) contain rich clinical information vital for cancer treatment decision making and research, yet reliably extracting structured oncology data remains challenging due to extensive variability, specialized terminology, and inconsistent document formats. Manual abstraction, although accurate, is prohibitively costly and unscalable. Existing automated approaches typically address narrow scenarios - either using synthetic datasets, restricting focus to document-level extraction, or isolating specific clinical variables (e.g., staging, biomarkers, histology) - and do not adequately handle patient-level synthesis across the large number of clinical documents containing contradictory information. In this study, we propose an agentic framework that systematically decomposes complex oncology data extraction into modular, adaptive tasks. Specifically, we use large language models (LLMs) as reasoning agents, equipped with context-sensitive retrieval and iterative synthesis capabilities, to exhaustively and comprehensively extract structured clinical variables from real-world oncology notes. Evaluated on a large-scale dataset of over 400,000 unstructured clinical notes and scanned PDF reports spanning 2,250 cancer patients, our method achieves an average F1-score of 0.93, with 100 out of 103 oncology-specific clinical variables exceeding 0.85, and critical variables (e.g., biomarkers and medications) surpassing 0.95. Moreover, integration of the agentic system into a data curation workflow resulted in 0.94 direct manual approval rate, significantly reducing annotation costs. To our knowledge, this constitutes the first exhaustive, end-to-end application of LLM-based agents for structured oncology data extraction at scale

Figures (9)

• Figure 1: Comparative analysis of HARMON-E versus traditional Named Entity Recognition (NER) approaches for medical document processing. A traditional NER system (lower left, red box) would incorrectly extract the planned end date (09/17/2020) without contextual understanding. In contrast, our agentic system (lower right, green box) performs multi-step reasoning: first identifying the planned treatment duration, then detecting the adverse event (pneumonitis), cross-referencing information about completed treatment cycles, and finally concluding that treatment actually ended on December 12, 2019 due to the adverse event—a conclusion impossible with traditional single-pass methods. (This is not real patient data and is for illustrative purposes only.)
• Figure 2: The HARMON-E transformation pipeline. Unstructured clinical documents from multiple sources (left) containing medications, biomarkers, staging information, and other oncology data are processed through the HARMON-E agentic extraction pipeline (center) to produce standardized, structured database entries (right). Each piece of clinically relevant information is extracted, validated, and organized into predefined entity-attribute pairs suitable for clinical research and decision support. The system processes heterogeneous inputs including progress notes, pathology reports, radiology impressions, and scanned PDFs, transforming approximately 180 documents per patient into comprehensive structured records.
• Figure 3: The tree structure illustrates the decomposition of a patient record into six primary entity categories (Medication, Biomarker, Diagnosis, Staging, Surgery, and Radiation), each containing multiple typed attributes. Representative examples from real oncology data are provided in italics below each attribute. This structured schema ensures consistency across heterogeneous clinical documentation and enables validation of extracted data against clinical standards. The complete model encompasses 16 entity types with 103 distinct attributes (subset shown for clarity).
• Figure 4: System workflow of HARMON-E. The system accepts raw, unstructured patient documents (HTML notes and/or scanned PDFs normalized as in Sec. \ref{['subsec:ingestion']}) and processes them through three main components: Retrievers, LLM Synthesizers, and Collators. Retrievers extract relevant segments for each targeted oncology entity; LLM Synthesizers transform these segments into structured attribute-value pairs; and Collators merge, validate, and resolve any dependencies among the extracted entities. The framework supports multiple pipeline strategies---from single-step to multi-step or topical extraction---accommodating a diverse range of real-world workflows. The final output is a patient-level structured data record spanning multiple oncology concepts (e.g., biomarkers, medications, TNM staging).
• Figure 5: Overview of Two Entity Alignment Methods.(A) Root-Based Alignment: An alignment is established only if both entities share the same root attribute (e.g., "medication" with value "Trastuzumab"), making other attributes (such as start dates) irrelevant for basic alignment. (B) Weighted Alignment: Each attribute (e.g., surgery_type, surgery_date, body_site) contributes a partial score based on a predefined weight. If the sum of matching attributes meets or exceeds a threshold (e.g., 0.9), the ground-truth and predicted entities are aligned.