Enhancing Trust and Safety in Digital Payments: An LLM-Powered Approach

TL;DR

The paper addresses rising scams in digital payments by introducing an LLM-driven scam detection framework targeted at UPI/GPay, combining a high-recall classifier with a reasoning-enabled digital assistant to aid human reviewers. Using Gemini-based LLMs, it introduces data preprocessing, feature engineering, and few-shot prompting/fine-tuning to achieve strong detection performance and interpretable explanations. The main contributions are a scalable LLM classifier achieving recall above 90%, a reasoning engine producing human-aligned explanations with 89% accuracy, and a generalizable workflow adaptable to other platforms and domains. The approach promises practical impact by reducing financial losses and reviewer workload, while outlining future work on on-device deployment, model distillation, RLHF, and cross-domain trust & safety applications.

Abstract

Digital payment systems have revolutionized financial transactions, offering unparalleled convenience and accessibility to users worldwide. However, the increasing popularity of these platforms has also attracted malicious actors seeking to exploit their vulnerabilities for financial gain. To address this challenge, robust and adaptable scam detection mechanisms are crucial for maintaining the trust and safety of digital payment ecosystems. This paper presents a comprehensive approach to scam detection, focusing on the Unified Payments Interface (UPI) in India, Google Pay (GPay) as a specific use case. The approach leverages Large Language Models (LLMs) to enhance scam classification accuracy and designs a digital assistant to aid human reviewers in identifying and mitigating fraudulent activities. The results demonstrate the potential of LLMs in augmenting existing machine learning models and improving the efficiency, accuracy, quality, and consistency of scam reviews, ultimately contributing to a safer and more secure digital payment landscape. Our evaluation of the Gemini Ultra model on curated transaction data showed a 93.33% accuracy in scam classification. Furthermore, the model demonstrated 89% accuracy in generating reasoning for these classifications. A promising fact, the model identified 32% new accurate reasons for suspected scams that human reviewers had not included in the review notes.

Figures (7)

• Figure 1: Input data for LLM Model (Gemini)
• Figure 2: Data Transformation Example
• Figure 3: LLM Fine-Tuning
• Figure 4: LLM Classifier with Reasoning: A Few-Shot Prompting Skeleton
• Figure 5: LLM Classifier Overall Performance & Sub-segments