Evaluating LLMs on Entity Disambiguation in Tables

TL;DR

This work proposes an extensive evaluation of four STI SOTA approaches: Alligator (formerly s-elbat), Dagobah, TURL, and TableLlama; the first two belong to the family of heuristic-based algorithms, while the others are respectively encoder-only and decoder-only Large Language Models (LLMs).

Abstract

Tables are crucial containers of information, but understanding their meaning may be challenging. Over the years, there has been a surge in interest in data-driven approaches based on deep learning that have increasingly been combined with heuristic-based ones. In the last period, the advent of \acf{llms} has led to a new category of approaches for table annotation. However, these approaches have not been consistently evaluated on a common ground, making evaluation and comparison difficult. This work proposes an extensive evaluation of four STI SOTA approaches: Alligator (formerly s-elbat), Dagobah, TURL, and TableLlama; the first two belong to the family of heuristic-based algorithms, while the others are respectively encoder-only and decoder-only Large Language Models (LLMs). We also include in the evaluation both GPT-4o and GPT-4o-mini, since they excel in various public benchmarks. The primary objective is to measure the ability of these approaches to solve the entity disambiguation task with respect to both the performance achieved on a common-ground evaluation setting and the computational and cost requirements involved, with the ultimate aim of charting new research paths in the field.

Figures (6)

• Figure 1: Architectures of TableLlama, TURL and Alligator.
• Figure 2: Per-approach average accuracy, elapsed time (min), and occupied memory (GB). The time reported is the elapsed time to link a mention, without considering the ER time. The occupied memory is the occupied GPU memory for TURL and TableLlama, while is the "Virtual Memory Size" for Alligator and Dagobah. {Approach}-FT represents the respective approach fine-tuned in MOOD setting (Sec. \ref{['sec:distribution-aware']}). "O.O.M." stands for Out-Of-Memory.
• Figure 3: Per-approach accuracy with respect to data typology, as defined in Section \ref{['sec:distribution-aware']}. The average is computed over datasets sharing the same data typology.
• Figure 4: Accuracies achieved by Alligator, GPT4o, TableLlama and TURL w.r.t. the number of candidates on the TURL-2K-red-LamAPI dataset.
• Figure 5: Accuracies achieved by GPT4o, TableLlama and TURL w.r.t. the presence of table's metadata on the TURL-2K-red-LamAPI dataset.