ToMMeR -- Efficient Entity Mention Detection from Large Language Models

TL;DR

The paper investigates whether entity mention boundaries are latent within LLM representations and can be recovered with a lightweight probe. It introduces ToMMeR, a sub-300K-parameter model that probes frozen LLM layers to identify valid spans using a simple matching mechanism and token-value cues, trained on broad Pile-NER data. Across 13 diverse NER benchmarks, ToMMeR achieves about 93% recall with approximately 92% precision when evaluated with an LLM judge, and cross-model analyses show Dice scores above 0.75 across architectures, indicating a shared boundary notion. Extending ToMMeR with a span-typing head yields near state-of-the-art full NER performance (80-87% F1) on standard benchmarks, demonstrating practical, schema-agnostic information extraction powered by early-layer signals in transformers.

Abstract

Identifying which text spans refer to entities -- mention detection -- is both foundational for information extraction and a known performance bottleneck. We introduce ToMMeR, a lightweight model (<300K parameters) probing mention detection capabilities from early LLM layers. Across 13 NER benchmarks, ToMMeR achieves 93\% recall zero-shot, with over 90\% precision using an LLM as a judge showing that ToMMeR rarely produces spurious predictions despite high recall. Cross-model analysis reveals that diverse architectures (14M-15B parameters) converge on similar mention boundaries (DICE >75\%), confirming that mention detection emerges naturally from language modeling. When extended with span classification heads, ToMMeR achieves near SOTA NER performance (80-87\% F1 on standard benchmarks). Our work provides evidence that structured entity representations exist in early transformer layers and can be efficiently recovered with minimal parameters.

Figures (14)

• Figure 1: We introduce ToMMeR: a lightweight probing model extracting emergent mention detection capabilities from early layers representations of any LLM backbone. Trained to generalize LLM annotated data, ToMMeR achieves high Zero Shot recall across a wide set of 13 NER benchmarks.
• Figure 2: The ToMMeR architecture. We extract the mention detection capabilities of any LLM backbone with less than $300$K additional parameters, without modifying the LLM parameters. We leverage Matching scores$m_{ij}$ between tokens $t_i$ and $t_j$ and individual values $v_{i}$ probed from LLM representations at layer $\ell$.
• Figure 3: DICE score between inference of ToMMeR trained on various LLMs on MultiNERD test (Results for GENIA are similar, see \ref{['app:AdditionnalFigs']}, fig. \ref{['fig:dice_Models_GENIA']}).
• Figure 4: F1 Score of ToMMeR Models —aggregated on the 13 benchmarks considered in this work- versus number of parameters of LLM backbone. We also plot the precision versus recall for all those models in appendix \ref{['fig:AllModels_Prec_Recall']}.
• Figure 5: DICE score between the sets of entities inferred on the MultiNERD (test), for ToMMeR models probing each layer of Llama3.2-1B. Results for GENIA are similar, and available in appendix, \ref{['fig:dice_Layers_GENIA']}.