Order from Chaos: Comparative Study of Ten Leading LLMs on Unstructured Data Categorization
Ariel Kamen
TL;DR
This work benchmarks ten major LLMs on unstructured text categorization against the IAB 2.2 taxonomy using a uniform 8,660-document dataset and a consistent zero-shot prompting strategy. Classic metrics (accuracy, precision, recall, F1) are modest across models, while LLM-specific costs and error modes reveal substantial hallucinatory and inflation tendencies; GPT-120B emerges as the strongest performer on cost-accuracy with the lowest hallucination rate. The authors also demonstrate that model scaling alone does not guarantee improvements, and that ensemble-based orchestration of multiple LLMs substantially boosts accuracy, curbs inflation, and can virtually eliminate hallucinations. The findings suggest that coordinated model ensembles, rather than sheer scale, may be the practical route to achieving or surpassing human-expert performance in large-scale text categorization within a fixed taxonomy.
Abstract
This study presents a comparative evaluation of ten state-of-the-art large language models (LLMs) applied to unstructured text categorization using the Interactive Advertising Bureau (IAB) 2.2 hierarchical taxonomy. The analysis employed a uniform dataset of 8,660 human-annotated samples and identical zero-shot prompts to ensure methodological consistency across all models. Evaluation metrics included four classic measures - accuracy, precision, recall, and F1-score - and three LLM-specific indicators: hallucination ratio, inflation ratio, and categorization cost. Results show that, despite their rapid advancement, contemporary LLMs achieve only moderate classic performance, with average scores of 34% accuracy, 42% precision, 45% recall, and 41% F1-score. Hallucination and inflation ratios reveal that models frequently overproduce categories relative to human annotators. Among the evaluated systems, Gemini 1.5/2.0 Flash and GPT 20B/120B offered the most favorable cost-to-performance balance, while GPT 120B demonstrated the lowest hallucination ratio. The findings suggest that scaling and architectural improvements alone do not ensure better categorization accuracy, as the task requires compressing rich unstructured text into a limited taxonomy - a process that challenges current model architectures. To address these limitations, a separate ensemble-based approach was developed and tested. The ensemble method, in which multiple LLMs act as independent experts, substantially improved accuracy, reduced inflation, and completely eliminated hallucinations. These results indicate that coordinated orchestration of models - rather than sheer scale - may represent the most effective path toward achieving or surpassing human-expert performance in large-scale text categorization.