Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Critical Role of Aspects in Measuring Document Similarity

Eftekhar Hossain, Tarnika Hazra, Ahatesham Bhuiyan, Santu Karmaker

TL;DR

AspectSim reframes document similarity as aspect-conditioned comparison, addressing interpretability gaps of holistic methods. It constructs a 26K-instance benchmark and demonstrates that GPT-4o prompting yields near-human agreement, approximately 0.90 correlation, while open-source LLMs require a two-stage extract-then-embed approach to reach substantial performance (approximately 0.58–0.59). The work analyzes open-model variability, baseline comparisons, and retrieval dynamics, highlighting the central role of explicit aspect conditioning and extraction quality for reliable similarity judgments. While progress with open models is promising, performance lags behind GPT-4o, motivating future work on retrieval-augmented approaches and downstream tasks. The study provides a reproducible framework and practical guidance for deploying aspect-conditioned similarity in NLP applications.

Abstract

We introduce ASPECTSIM, a simple and interpretable framework that requires conditioning document similarity on an explicitly specified aspect, which is different from the traditional holistic approach in measuring document similarity. Experimenting with a newly constructed benchmark of 26K aspect-document pairs, we found that ASPECTSIM, when implemented with direct GPT-4o prompting, achieves substantially higher human-machine agreement ($\approx$80% higher) than the same for holistic similarity without explicit aspects. These findings underscore the importance of explicitly accounting for aspects when measuring document similarity and highlight the need to revise standard practice. Next, we conducted a large-scale meta-evaluation using 16 smaller open-source LLMs and 9 embedding models with a focus on making ASPECTSIM accessible and reproducible. While directly prompting LLMs to produce ASPECTSIM scores turned out be ineffective (20-30% human-machine agreement), a simple two-stage refinement improved their agreement by $\approx$140%. Nevertheless, agreement remains well below that of GPT-4o-based models, indicating that smaller open-source LLMs still lag behind large proprietary models in capturing aspect-conditioned similarity.

The Critical Role of Aspects in Measuring Document Similarity

TL;DR

AspectSim reframes document similarity as aspect-conditioned comparison, addressing interpretability gaps of holistic methods. It constructs a 26K-instance benchmark and demonstrates that GPT-4o prompting yields near-human agreement, approximately 0.90 correlation, while open-source LLMs require a two-stage extract-then-embed approach to reach substantial performance (approximately 0.58–0.59). The work analyzes open-model variability, baseline comparisons, and retrieval dynamics, highlighting the central role of explicit aspect conditioning and extraction quality for reliable similarity judgments. While progress with open models is promising, performance lags behind GPT-4o, motivating future work on retrieval-augmented approaches and downstream tasks. The study provides a reproducible framework and practical guidance for deploying aspect-conditioned similarity in NLP applications.

Abstract

We introduce ASPECTSIM, a simple and interpretable framework that requires conditioning document similarity on an explicitly specified aspect, which is different from the traditional holistic approach in measuring document similarity. Experimenting with a newly constructed benchmark of 26K aspect-document pairs, we found that ASPECTSIM, when implemented with direct GPT-4o prompting, achieves substantially higher human-machine agreement (80% higher) than the same for holistic similarity without explicit aspects. These findings underscore the importance of explicitly accounting for aspects when measuring document similarity and highlight the need to revise standard practice. Next, we conducted a large-scale meta-evaluation using 16 smaller open-source LLMs and 9 embedding models with a focus on making ASPECTSIM accessible and reproducible. While directly prompting LLMs to produce ASPECTSIM scores turned out be ineffective (20-30% human-machine agreement), a simple two-stage refinement improved their agreement by 140%. Nevertheless, agreement remains well below that of GPT-4o-based models, indicating that smaller open-source LLMs still lag behind large proprietary models in capturing aspect-conditioned similarity.
Paper Structure (57 sections, 6 equations, 10 figures, 7 tables, 1 algorithm)

This paper contains 57 sections, 6 equations, 10 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Aspect-Based Document Similarity.
  4. Aspect-Guided Evidence Extraction.
  5. Method
  6. Dataset Construction
  7. Data Source Selection
  8. GPT-4o-Based Dataset Curation
  9. Task 1: Aspect Identification.
  10. Task 2: Aspect-Conditioned Extraction.
  11. Task 3: Silver Standard Ground Truth Generation.
  12. Negative Instance Generation
  13. Data Quality Validation by Humans
  14. Does Explicit Aspect Indeed Improve Similairy Assessment?
  15. Implementing AspectSim Metric
  16. ...and 42 more sections

Figures (10)

  • Figure 1: Illustration of how document similarity might differ when it is viewed through multiple aspects.
  • Figure 2: AspectSim performance across five datasets under Sentence-level and Span-level settings for different model-size ranges. Here, only the highest-performing embedding per dataset is shown.
  • Figure 3: AspectSim performance across aspect lengths grouped by token count.
  • Figure 4: AspectSim retrieval accuracy across various LLMs at the sentence-level. BM denotes correctly retrieved aspect-relevant units. F-MM, S-MM, and B-MM indicate mismatched extractions for the first, second, and both documents, respectively. F-E, S-E, and B-E represent missed extractions (empty outputs) in the corresponding documents.
  • Figure 5: Effect of document length and sentence position on sentence-level retrieval success across model sizes.
  • ...and 5 more figures