Behavioural Effects of Agentic Messaging: A Case Study on a Financial Service Application

TL;DR

The paper evaluates a case study of agentic personalisation in a financial service CRM during the 2025 tax filing season. It frames agentic messaging as sequential decision-making with Bayesian Thompson sampling and a Difference-in-Differences estimator to assess incremental impact. In a large randomized trial with 6.4 million users, agentic messaging reduced unsubscribes by 21% and accelerated conversions several weeks before the deadline, without increasing message volume. The findings suggest that adaptive, user-level decision-making systems can improve retention and operational efficiency, with potential generalisability across industries.

Abstract

Marketing and product personalisation provide a prominent and visible use-case for the application of Information Retrieval methods across several business domains. Recently, agentic approaches to these problems have been gaining traction. This work evaluates the behavioural and retention effects of agentic personalisation on a financial service application's customer communication system during a 2025 national tax filing period. Through a two month-long randomised controlled trial, we compare an agentic messaging approach against a business-as-usual (BAU) rule-based campaign system, focusing on two primary outcomes: unsubscribe behaviour and conversion timing. Empirical results show that agent-led messaging reduced unsubscribe events by 21\% ($\pm 0.01$) relative to BAU and increased early filing behaviour in the weeks preceding the national deadline. These findings demonstrate how adaptive, user-level decision-making systems can modulate engagement intensity whilst improving long-term retention indicators.

Figures (2)

• Figure 1: Empirical results for unsubscription rates in the A/B-test.
• Figure 2: Relative increase in event frequencies in the A/B-test.