Robust LLM-based Column Type Annotation via Prompt Augmentation with LoRA Tuning

TL;DR

This work tackles the instability of large-language-model–driven column type annotation (CTA) caused by prompt sensitivity and distribution shifts. It introduces a parameter-efficient framework that combines prompt augmentation with Low-Rank Adaptation (LoRA) to train lightweight adapters on augmented prompts while keeping the backbone frozen. Across VizNet and SOTAB, the approach yields more stable and higher weighted F1 scores than single-prompt-template fine-tuning, and achieves data efficiency by matching full-data performance with only a fraction of labeled columns. The results demonstrate that prompt augmentation is a robust, model-agnostic strategy for practical CTA systems, with potential extensions to unlabeled data and semi-supervised learning.

Abstract

Column Type Annotation (CTA) is a fundamental step towards enabling schema alignment and semantic understanding of tabular data. Existing encoder-only language models achieve high accuracy when fine-tuned on labeled columns, but their applicability is limited to in-domain settings, as distribution shifts in tables or label spaces require costly re-training from scratch. Recent work has explored prompting generative large language models (LLMs) by framing CTA as a multiple-choice task, but these approaches face two key challenges: (1) model performance is highly sensitive to subtle changes in prompt wording and structure, and (2) annotation F1 scores remain modest. A natural extension is to fine-tune large language models. However, fully fine-tuning these models incurs prohibitive computational costs due to their scale, and the sensitivity to prompts is not eliminated. In this paper, we present a parameter-efficient framework for CTA that trains models over prompt-augmented data via Low-Rank Adaptation (LoRA). Our approach mitigates sensitivity to prompt variations while drastically reducing the number of necessary trainable parameters, achieving robust performance across datasets and templates. Experimental results on recent benchmarks demonstrate that models fine-tuned with our prompt augmentation strategy maintain stable performance across diverse prompt patterns during inference and yield higher weighted F1 scores than those fine-tuned on a single prompt template. These results highlight the effectiveness of parameter-efficient training and augmentation strategies in developing practical and adaptable CTA systems.

Figures (8)

• Figure 1: An example of the prompt-based Column Type Annotation (CTA) task. Given a table with unlabeled columns, column entries are presented along with candidate semantic labels in a prompt to a language model. The model predicts the proper semantic type, which in this case is "Location".
• Figure 2: Illustration of Low-Rank Adaptation (LoRA) hu2022lora. Instead of updating the full matrix $W$, LoRA freezes $W$ and learns two smaller matrices $A$ and $B$ with rank $r \ll d$. This reduces the number of trainable parameters by orders of magnitude while retaining the pre-trained knowledge in $W$.
• Figure 3: Overview of our parameter-efficient fine-tuning pipeline for prompt-based column type annotation (CTA). Given a column, we instantiate the three template patterns to form concrete prompts with different column entry sampling strategies: Random Sampling, Shortest Sampling, and Archetype Sampling, each encoding a different contextual view. These prompts are passed into a frozen pre-trained language model augmented with a trainable LoRA Adapter. The model's prediction (Response) is compared with the ground-truth label ("Organization") via a cross-entropy loss, whose gradients are backpropagated to update the LoRA adapter. Only the weights in the LoRA adapter are updated during fine-tuning, leaving the base model untouched. This design enables efficient adaptation and prompt-based augmentation without modifying the underlying model.
• Figure 4: Three prompt patterns identified in prior prompt-based CTA literature. All patterns are populated with the values from the first column in Figure \ref{['fig:running-example']}. $t_i$, $s_c$ and $s_\ell$ are the prompt components as defined in Section \ref{['subsubsec:cta_prompt_comp']}.
• Figure 5: Weighted F1 scores of FLAN-T5 Base LoRA fine-tuned with and without prompt augmentation on the SOTAB dataset.