Understanding and Mitigating Risks of Generative AI in Financial Services

TL;DR

This paper argues that generic GenAI safety taxonomies and guardrails are insufficient for the financial services domain due to its complex sociotechnical context. It develops a domain-specific AI content safety taxonomy anchored to three stakeholder groups (buy-side, sell-side, vendors) and relevant regulatory considerations, and then empirically evaluates off-the-shelf guardrails against this taxonomy. Using red-teaming data (10,400 inputs, 7,340 outputs) and a normal-course dataset, the study demonstrates a persistent safety gap: high precision but poor recall, especially for domain-specific risks like Financial Services Impartiality and Social Media Headline Risk. The authors advocate a holistic, multi-layer governance approach, domain-adapted risk frameworks, and active involvement of academics to close the safety gap and make GenAI deployments in finance safer and compliant with industry rules.

Abstract

To responsibly develop Generative AI (GenAI) products, it is critical to define the scope of acceptable inputs and outputs. What constitutes a "safe" response is an actively debated question. Academic work puts an outsized focus on evaluating models by themselves for general purpose aspects such as toxicity, bias, and fairness, especially in conversational applications being used by a broad audience. In contrast, less focus is put on considering sociotechnical systems in specialized domains. Yet, those specialized systems can be subject to extensive and well-understood legal and regulatory scrutiny. These product-specific considerations need to be set in industry-specific laws, regulations, and corporate governance requirements. In this paper, we aim to highlight AI content safety considerations specific to the financial services domain and outline an associated AI content risk taxonomy. We compare this taxonomy to existing work in this space and discuss implications of risk category violations on various stakeholders. We evaluate how existing open-source technical guardrail solutions cover this taxonomy by assessing them on data collected via red-teaming activities. Our results demonstrate that these guardrails fail to detect most of the content risks we discuss.