Interpreting LLMs as Credit Risk Classifiers: Do Their Feature Explanations Align with Classical ML?

TL;DR

This study probes whether zero-shot large language models can serve as classifiers for structured financial data and how their explanations compare with a strong classical baseline. By evaluating zero-shot LLMs against LightGBM on a real loan-default task and auditing explanations with SHAP, the authors assess both predictive performance and faithfulness of reasoning. They find that while LLMs identify key risk indicators, their feature attributions and self-explanations diverge from SHAP and from LightGBM's patterns, and ensembling yields limited gains. The results underscore the need for explainability audits and human-in-the-loop oversight before deploying LLMs in high-stakes finance, suggesting LightGBM remains the more reliable baseline absent targeted LLM fine-tuning and robust reliability assessments.

Abstract

Large Language Models (LLMs) are increasingly explored as flexible alternatives to classical machine learning models for classification tasks through zero-shot prompting. However, their suitability for structured tabular data remains underexplored, especially in high-stakes financial applications such as financial risk assessment. This study conducts a systematic comparison between zero-shot LLM-based classifiers and LightGBM, a state-of-the-art gradient-boosting model, on a real-world loan default prediction task. We evaluate their predictive performance, analyze feature attributions using SHAP, and assess the reliability of LLM-generated self-explanations. While LLMs are able to identify key financial risk indicators, their feature importance rankings diverge notably from LightGBM, and their self-explanations often fail to align with empirical SHAP attributions. These findings highlight the limitations of LLMs as standalone models for structured financial risk prediction and raise concerns about the trustworthiness of their self-generated explanations. Our results underscore the need for explainability audits, baseline comparisons with interpretable models, and human-in-the-loop oversight when deploying LLMs in risk-sensitive financial environments.

Figures (6)

• Figure 1: Comparative Explainable AI Framework: Classical ML vs. LLMs. The dataset is processed through two paradigms: (i) a structured LightGBM model and (ii) a zero-shot LLM using natural language prompts. Both generate probability predictions, analyzed individually and in an ensemble. Explainability is assessed via SHAP (for both) and LLM self-explanations, evaluating their alignment.
• Figure 2: ROC and Precision-Recall curves comparing the performance of zero-shot LLMs and LightGBM on the loan classification task. LightGBM consistently outperforms individual LLMs with Gemma-2-9b showing the most promising result out of the LLMs.
• Figure 3: SHAP feature importance comparison between LightGBM and LLMs. Despite being in a zero-shot setting, LLMs identify a remarkably similar set of key financial features as LightGBM, though with differences in feature weighting and distribution.
• Figure 4: SHAP summary plots comparing feature importance distributions for LLMs and LightGBM. LightGBM shows a more structured reliance on key financial indicators, while LLMs exhibit more dispersed and lower-magnitude SHAP values, indicating weaker feature dependencies.
• Figure 5: SHAP Feature dependence plots and LLM self-explanations for the feature DTI.