Cost-Efficient Prompt Engineering for Unsupervised Entity Resolution

TL;DR

The paper investigates cost-efficient prompting strategies for unsupervised entity resolution (ER) using GPT-3.5 across two real-world e-commerce datasets. It defines six prompt patterns (including single-attribute, multi-attribute with/without a persona, JSON formatting, similarity scoring, and few-shot) and evaluates their performance and cost. Key findings show that simpler prompts often achieve competitive F-measures, JSON-formatted prompts can degrade performance, and there is non-negligible inter-method disagreement, highlighting the importance of prompt design for LLM-based ER. The work provides practical guidance on selecting prompts for cost-conscious ER deployments and contributes data for replication and further analysis, while noting limitations around blocking integration and domain-specific challenges.

Abstract

Entity Resolution (ER) is the problem of semi-automatically determining when two entities refer to the same underlying entity, with applications ranging from healthcare to e-commerce. Traditional ER solutions required considerable manual expertise, including domain-specific feature engineering, as well as identification and curation of training data. Recently released large language models (LLMs) provide an opportunity to make ER more seamless and domain-independent. However, it is also well known that LLMs can pose risks, and that the quality of their outputs can depend on how prompts are engineered. Unfortunately, a systematic experimental study on the effects of different prompting methods for addressing unsupervised ER, using LLMs like ChatGPT, has been lacking thus far. This paper aims to address this gap by conducting such a study. We consider some relatively simple and cost-efficient ER prompt engineering methods and apply them to ER on two real-world datasets widely used in the community. We use an extensive set of experimental results to show that an LLM like GPT3.5 is viable for high-performing unsupervised ER, and interestingly, that more complicated and detailed (and hence, expensive) prompting methods do not necessarily outperform simpler approaches. We provide brief discussions on qualitative and error analysis, including a study of the inter-consistency of different prompting methods to determine whether they yield stable outputs. Finally, we consider some limitations of LLMs when applied to ER.

Figures (12)

• Figure 1: An illustrative example of products from Google Product Search offering the same product from different sources
• Figure 2: An illustrative example of a typical ER workflow, but with an LLM used as the similarity (or 'matching') function. Each entity collection represents a structured ER dataset, with individual entities represented using colored boxes. As explained in Section \ref{['sec:background']}, blocking is first applied to cluster approximately similar entities into blocks, in order to mitigate the quadratic complexity of comparing all pairs of entities. Only entities sharing a block are paired and presented to the LLM for making a final decision on whether they match (yes) or not (no).
• Figure 3: A text description of the persona describing the conceptual role of GPT-3.5 when performing ER. Since this study uses a dataset in the product domain, we set up the role of GPT-3.5 as an expert on product classification, with an emphasis on using related knowledge from multiple product domains to resolve the pair.
• Figure 4: An illustrative example of the single-attr prompt pattern. The example includes two records for comparison: both are versions of 'Apple Final Cut Studio 2'. The records information only contains product titles (highlighted in blue). The task instruction asks GPT-3.5 to determine if the records refer to identical products, considering their similarities and differences. The GPT-3.5 response shows that the records are identical and explains the reasoning behind the decision, along with an associated confidence level.
• Figure 5: An illustrative example of the multi-attr and no-persona prompt patterns, comparing records with multiple attributes concatenated (highlighted in blue). As with the single-attr pattern, GPT-3.5 is tasked with evaluating whether the detailed product records, now including additional attributes like manufacturer, description, and price, refer to the same underlying product. The GPT-3.5 response format remains the same as the single-attr pattern.