Approaches to Semantic Textual Similarity in Slovak Language: From Algorithms to Transformers

TL;DR

This work tackles semantic textual similarity for Slovak, an under‑resourced language, by performing a comprehensive comparison of traditional STS algorithms, supervised ML models that use traditional outputs as features tuned with Artificial Bee Colony optimization, and several third‑party tools including OpenAI embeddings, GPT‑4, NLPCloud, and a fine‑tuned SlovakBERT. Term‑based traditional methods emerge as the strongest among non‑embeddings, while modern deep‑learning tools consistently outperform traditional approaches, with NLPCloud Paraphrase MPNet Base V2 achieving the highest scores among the evaluated third‑party systems and GPT‑4 delivering the strongest single‑model performance. Fine‑tuning SlovakBERT yields competitive results comparable to OpenAI embeddings at a lower cost, illustrating the value of language‑specific resources. The study provides practical guidance on method selection for Slovak STS, highlighting trade‑offs between accuracy, cost, and resource availability, and points to the need for higher‑quality linguistic resources for low‑resource languages.

Abstract

Semantic textual similarity (STS) plays a crucial role in many natural language processing tasks. While extensively studied in high-resource languages, STS remains challenging for under-resourced languages such as Slovak. This paper presents a comparative evaluation of sentence-level STS methods applied to Slovak, including traditional algorithms, supervised machine learning models, and third-party deep learning tools. We trained several machine learning models using outputs from traditional algorithms as features, with feature selection and hyperparameter tuning jointly guided by artificial bee colony optimization. Finally, we evaluated several third-party tools, including fine-tuned model by CloudNLP, OpenAI's embedding models, GPT-4 model, and pretrained SlovakBERT model. Our findings highlight the trade-offs between different approaches.

Figures (1)

• Figure 1: Overview of the evaluation pipeline, including preprocessing, optimization, model training, and third-party tool evaluation.