Taxonomy-Aligned Risk Extraction from 10-K Filings with Autonomous Improvement Using LLMs

TL;DR

This work tackles scalable extraction of structured risk factors from lengthy 10-K disclosures by enforcing alignment to a predefined taxonomy. It introduces a three-stage pipeline that first uses an LLM to extract risks with supporting quotes, then maps these quotes to taxonomy categories via embedding-based similarity, and finally employs an LLM-as-judge to validate mappings and suppress spurious assignments. A novel autonomous taxonomy refinement workflow uses evaluation feedback to identify problematic categories, diagnose failure patterns, generate refinements, and validate improvements, achieving a 104.7% gain in embedding separation for a pharmaceutical-approval category. Empirical validation on 2024 S&P 500 filings yields 10,688 validated risk factors and demonstrates economically meaningful structure via industry clustering: same-industry risk profiles are 63% more similar than cross-industry ones (AUC up to 0.822 at finer industry granularity). The approach generalizes to any domain requiring taxonomy-aligned extraction from unstructured text and enables continuous, autonomous taxonomy maintenance as more documents are processed.

Abstract

We present a methodology for extracting structured risk factors from corporate 10-K filings while maintaining adherence to a predefined hierarchical taxonomy. Our three-stage pipeline combines LLM extraction with supporting quotes, embedding-based semantic mapping to taxonomy categories, and LLM-as-a-judge validation that filters spurious assignments. To evaluate our approach, we extract 10,688 risk factors from S&P 500 companies and examine risk profile similarity across industry clusters. Beyond extraction, we introduce autonomous taxonomy maintenance where an AI agent analyzes evaluation feedback to identify problematic categories, diagnose failure patterns, and propose refinements, achieving 104.7% improvement in embedding separation in a case study. External validation confirms the taxonomy captures economically meaningful structure: same-industry companies exhibit 63% higher risk profile similarity than cross-industry pairs (Cohen's d=1.06, AUC 0.82, p<0.001). The methodology generalizes to any domain requiring taxonomy-aligned extraction from unstructured text, with autonomous improvement enabling continuous quality maintenance and enhancement as systems process more documents.

Figures (4)

• Figure 1: Three-stage pipeline for taxonomy-aligned risk extraction. Stage 1 uses an LLM to extract risk factors with supporting quotes from raw text. Stage 2 maps extracted risks to taxonomy categories using embedding-based semantic similarity. Stage 3 employs an LLM judge to validate mappings and filter spurious assignments, while low-quality mappings provide feedback for continuous taxonomy improvement.
• Figure 2: Risk profile similarity distributions for same-industry versus different-industry company pairs using weighted similarity. Companies sharing 2-digit SIC codes (blue, 5,263 pairs) exhibit substantially higher similarity than companies in different industries (red, 101,228 pairs). The clear distributional separation demonstrates that extracted risk profiles naturally capture sectoral information despite no industry information in the taxonomy mapping process.
• Figure 3: ROC curves for predicting industry membership from weighted risk similarity across three SIC code granularities. Finer industry definitions (4-digit SIC, AUC = 0.822) produce stronger clustering than broad sectors (2-digit, AUC = 0.733), demonstrating that extracted risk profiles capture industry-specific patterns at multiple levels of specificity. All curves substantially outperform random classification (diagonal, AUC = 0.5).
• Figure 4: Risk category prevalence for Depository Institutions (SIC 60, $n=12$) versus overall S&P 500 population. Bars show the percentage of companies tagged with each risk. Banks exhibit systematic overrepresentation of finance-specific risks: 83% have interest rate risk versus 22% overall, 67% have capital requirements versus 3% overall. Risks are sorted by overall S&P prevalence (highest at top) to emphasize sector-specific enrichment.