Are LLM-based methods good enough for detecting unfair terms of service?

TL;DR

This paper tackles the challenge of detecting unfair terms in long terms of service and privacy policies using large language models. It introduces the ToS-Busters dataset, consisting of 220 policies assessed with 12 targeted questions, and ground-truth answers from PrivacySpy, enabling a document-level question-answering benchmark. Through experiments with multiple open-source and commercial LLMs, including GPT-4-turbo, the study finds that while GPT-4-turbo achieves the best average accuracy (~53%), all models perform only modestly above random, highlighting the difficulty of long-document ToS analysis and the need for further improvements in summarization and prompting. The work provides a公开 benchmark and a roadmap for enhancing user-facing tools to identify dubious clauses in online contracts, with future directions focusing on model improvements, summarization effects, and prompt design.

Abstract

Countless terms of service (ToS) are being signed everyday by users all over the world while interacting with all kinds of apps and websites. More often than not, these online contracts spanning double-digit pages are signed blindly by users who simply want immediate access to the desired service. What would normally require a consultation with a legal team, has now become a mundane activity consisting of a few clicks where users potentially sign away their rights, for instance in terms of their data privacy, to countless online entities/companies. Large language models (LLMs) are good at parsing long text-based documents, and could potentially be adopted to help users when dealing with dubious clauses in ToS and their underlying privacy policies. To investigate the utility of existing models for this task, we first build a dataset consisting of 12 questions applied individually to a set of privacy policies crawled from popular websites. Thereafter, a series of open-source as well as commercial chatbots such as ChatGPT, are queried over each question, with the answers being compared to a given ground truth. Our results show that some open-source models are able to provide a higher accuracy compared to some commercial models. However, the best performance is recorded from a commercial chatbot (ChatGPT4). Overall, all models perform only slightly better than random at this task. Consequently, their performance needs to be significantly improved before they can be adopted at large for this purpose.