Evaluating Knowledge Generation and Self-Refinement Strategies for LLM-based Column Type Annotation
Keti Korini, Christian Bizer
TL;DR
This work systematically evaluates knowledge generation and self-refinement strategies for LLM-based column type annotation under low-data, metadata-scarce settings. It compares knowledge-prompted label definitions, self-refinement, self-correction, and fine-tuning across three CTA datasets and four models, measuring both F1 performance and token-cost efficiency. Key findings include: no single best strategy across all model/dataset pairs; knowledge-generation prompting and refined label-definitions yield consistent F1 gains; self-refinement provides additional improvements on many setups, while self-correction is generally less beneficial; fine-tuning offers substantial gains for large-scale data and can be even more cost-efficient than prompting in high-volume use cases. The results inform practical deployment by balancing accuracy against token usage and costs, especially highlighting when to prefer fine-tuning versus prompting-based refinement for CTA tasks.
Abstract
Understanding the semantics of columns in relational tables is an important pre-processing step for indexing data lakes in order to provide rich data search. An approach to establishing such understanding is column type annotation (CTA) where the goal is to annotate table columns with terms from a given vocabulary. This paper experimentally compares different knowledge generation and self-refinement strategies for LLM-based column type annotation. The strategies include using LLMs to generate term definitions, error-based refinement of term definitions, self-correction, and fine-tuning using examples and term definitions. We evaluate these strategies along two dimensions: effectiveness measured as F1 performance and efficiency measured in terms of token usage and cost. Our experiments show that the best performing strategy depends on the model/dataset combination. We find that using training data to generate label definitions outperforms using the same data as demonstrations for in-context learning for two out of three datasets using OpenAI models. The experiments further show that using the LLMs to refine label definitions brings an average increase of 3.9% F1 in 10 out of 12 setups compared to the performance of the non-refined definitions. Combining fine-tuned models with self-refined term definitions results in the overall highest performance, outperforming zero-shot prompting fine-tuned models by at least 3% in F1 score. The costs analysis shows that while reaching similar F1 score, self-refinement via prompting is more cost efficient for use cases requiring smaller amounts of tables to be annotated while fine-tuning is more efficient for large amounts of tables.