Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating Transparent Models, LLMs, and Practitioner-in-the-Loop: A Case of Nonprofit Program Evaluation

Ji Ma, Albert Casella

TL;DR

Public and nonprofit organizations face barriers adopting AI due to model opacity and a lack of case-level, actionable guidance. The paper tests a practitioner-in-the-loop workflow that combines a transparent decision-tree predictor with an LLM to generate grounded, case-specific explanations, using data from a college-scholarship program. Findings show strong predictive accuracy comparable to black-box approaches, and that incorporating a program knowledge base into LLM prompts enhances perceived safety and fairness, underscoring the value of domain expertise in responsible AI deployment. The study offers a practical blueprint for integrating interpretable models, LLM-assisted explanations, and practitioner input to support targeted interventions and trustworthy evaluation in public-sector contexts.

Abstract

Public and nonprofit organizations often hesitate to adopt AI tools because most models are opaque even though standard approaches typically analyze aggregate patterns rather than offering actionable, case-level guidance. This study tests a practitioner-in-the-loop workflow that pairs transparent decision-tree models with large language models (LLMs) to improve predictive accuracy, interpretability, and the generation of practical insights. Using data from an ongoing college-success program, we build interpretable decision trees to surface key predictors. We then provide each tree's structure to an LLM, enabling it to reproduce case-level predictions grounded in the transparent models. Practitioners participate throughout feature engineering, model design, explanation review, and usability assessment, ensuring that field expertise informs the analysis at every stage. Results show that integrating transparent models, LLMs, and practitioner input yields accurate, trustworthy, and actionable case-level evaluations, offering a viable pathway for responsible AI adoption in the public and nonprofit sectors.

Integrating Transparent Models, LLMs, and Practitioner-in-the-Loop: A Case of Nonprofit Program Evaluation

TL;DR

Public and nonprofit organizations face barriers adopting AI due to model opacity and a lack of case-level, actionable guidance. The paper tests a practitioner-in-the-loop workflow that combines a transparent decision-tree predictor with an LLM to generate grounded, case-specific explanations, using data from a college-scholarship program. Findings show strong predictive accuracy comparable to black-box approaches, and that incorporating a program knowledge base into LLM prompts enhances perceived safety and fairness, underscoring the value of domain expertise in responsible AI deployment. The study offers a practical blueprint for integrating interpretable models, LLM-assisted explanations, and practitioner input to support targeted interventions and trustworthy evaluation in public-sector contexts.

Abstract

Public and nonprofit organizations often hesitate to adopt AI tools because most models are opaque even though standard approaches typically analyze aggregate patterns rather than offering actionable, case-level guidance. This study tests a practitioner-in-the-loop workflow that pairs transparent decision-tree models with large language models (LLMs) to improve predictive accuracy, interpretability, and the generation of practical insights. Using data from an ongoing college-success program, we build interpretable decision trees to surface key predictors. We then provide each tree's structure to an LLM, enabling it to reproduce case-level predictions grounded in the transparent models. Practitioners participate throughout feature engineering, model design, explanation review, and usability assessment, ensuring that field expertise informs the analysis at every stage. Results show that integrating transparent models, LLMs, and practitioner input yields accurate, trustworthy, and actionable case-level evaluations, offering a viable pathway for responsible AI adoption in the public and nonprofit sectors.
Paper Structure (24 sections, 6 figures, 5 tables)

This paper contains 24 sections, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Data Source
  4. Measures
  5. Predictive and Explanatory Models
  6. Predictive Model with Transparency: Decision Tree
  7. Case-Level Prediction and Interpretation: Large Language Model
  8. Evaluation
  9. Model Performance Metrics
  10. Practitioner-in-the-Loop
  11. Results
  12. Descriptive Statistics
  13. Model Performance
  14. Case‐Level Prediction and Interpretation
  15. Usability Assessment
  16. ...and 9 more sections

Figures (6)

  • Figure 1: Practitioner-in-the-Loop Workflow
  • Figure 2: Usability Test Results for LLM-Generated Explanations
  • Figure 3: Usability Test Results: With vs. Without Program Knowledge
  • Figure B4: LLM Prompt Template for Case-Level Explanations (Without Knowledge Base)
  • Figure B5: LLM Prompt Template for Case-Level Explanations (With Knowledge Base for In-Context Learning)
  • ...and 1 more figures