FicSim: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fiction

TL;DR

FicSim addresses the lack of robust long-form literary STS benchmarks by constructing a multi-axis dataset drawn from long-form fanfiction with author consent. The authors derive gold-standard similarity across 12 categories from detailed fanfiction tags and compute category-specific similarity using Gemini embeddings, validating these scores with expert annotations. Across a suite of open and API-based embedding methods, they find that current models struggle to capture fine-grained literary semantics and tend to over-index on superficial cues such as author identity or fandom. The study also evaluates long-context strategies like sliding windows and category-specific prompts, concluding that these adjustments offer limited gains. Overall, FicSim highlights a substantial gap between current embedding capabilities and the needs of computational literary studies, providing a practical, rights-respecting resource to guide model selection and future methodological development in DH tasks.

Abstract

As language models become capable of processing increasingly long and complex texts, there has been growing interest in their application within computational literary studies. However, evaluating the usefulness of these models for such tasks remains challenging due to the cost of fine-grained annotation for long-form texts and the data contamination concerns inherent in using public-domain literature. Current embedding similarity datasets are not suitable for evaluating literary-domain tasks because of a focus on coarse-grained similarity and primarily on very short text. We assemble and release FICSIM, a dataset of long-form, recently written fiction, including scores along 12 axes of similarity informed by author-produced metadata and validated by digital humanities scholars. We evaluate a suite of embedding models on this task, demonstrating a tendency across models to focus on surface-level features over semantic categories that would be useful for computational literary studies tasks. Throughout our data-collection process, we prioritize author agency and rely on continual, informed author consent.

Figures (6)

• Figure 1: Similarity between literary texts can be defined along many dimensions. Computational literary studies scholars generally seek to measure specific, semantic types of similarity such as similarity in tone, but embedding models over-index on more obvious features such as the author's writing style.
• Figure 2: Example of fic tagging and metadata on AO3. Colored annotations mark data that inform similarity scores.
• Figure 2: Types of similarity and example tags that align with each category. Some of the above tags were placed into multiple categories within our dataset, but they nonetheless serve as useful references for the particular category they represent above.
• Figure 3: Story lengths in FicSim range from 10k to >400k words.
• Figure 4: Sample comment on fanfiction