IdentifyMe: A Challenging Long-Context Mention Resolution Benchmark for LLMs

TL;DR

This paper introduces IdentifyMe, a challenging long-context coreference resolution benchmark reformulated as MCQs to better assess LLM referential understanding. It constructs MCQ items from LitBank and FantasyCoref using a two-step mention selection and generates representative phrases for entity clusters, with human validation to ensure clarity. Experiments compare closed- and open-source LLMs, revealing a sizable gap between frontier models (e.g., GPT-4o with 81.9% accuracy) and sub-10B open models, particularly on pronominal and None-of-the-Above cases. Error analysis highlights nested mentions and NoA detection as core bottlenecks, guiding future work toward more robust long-context referential reasoning.

Abstract

Recent evaluations of LLMs on coreference resolution have revealed that traditional output formats and evaluation metrics do not fully capture the models' referential understanding. To address this, we introduce IdentifyMe, a new benchmark for mention resolution presented in a multiple-choice question (MCQ) format, commonly used for evaluating LLMs. IdentifyMe features long narratives and employs heuristics to exclude easily identifiable mentions, creating a more challenging task. The benchmark also consists of a curated mixture of different mention types and corresponding entities, allowing for a fine-grained analysis of model performance. We evaluate both closed- and open source LLMs on IdentifyMe and observe a significant performance gap (20-30%) between the state-of-the-art sub-10B open models vs. closed ones. We observe that pronominal mentions, which have limited surface information, are typically much harder for models to resolve than nominal mentions. Additionally, we find that LLMs often confuse entities when their mentions overlap in nested structures. The highest-scoring model, GPT-4o, achieves 81.9% accuracy, highlighting the strong referential capabilities of state-of-the-art LLMs while also indicating room for further improvement.

Figures (5)

• Figure 1: Sample instance from the validation set of IdentifyMe. The mention of interest is highlighted in the text. The answer options include frequently occurring entities in the text, and None of the Above.
• Figure 2: An error by GPT-4o in resolving a nested mention where the model incorrectly resolves his artless victim to the entity referred to by his i.e. M. Capoul.
• Figure 3: A sample error made by GPT-4o where Sherlock Holmes and Dr. John Watson are engaged in a dialog. The instance is particularly hard because the dialog speakers are not marked and need to be inferred.
• Figure 4: Sample instance from IdentifyMe that both GPT-4o and Llama-3.1 get right.
• Figure 5: Sample instance from IdentifyMe that both GPT-4o and Llama-3.1 get right.