Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Developing and evaluating a chatbot to support maternal health care

Smriti Jha, Vidhi Jain, Jianyu Xu, Grace Liu, Sowmya Ramesh, Jitender Nagpal, Gretchen Chapman, Benjamin Bellows, Siddhartha Goyal, Aarti Singh, Bryan Wilder

Abstract

The ability to provide trustworthy maternal health information using phone-based chatbots can have a significant impact, particularly in low-resource settings where users have low health literacy and limited access to care. However, deploying such systems is technically challenging: user queries are short, underspecified, and code-mixed across languages, answers require regional context-specific grounding, and partial or missing symptom context makes safe routing decisions difficult. We present a chatbot for maternal health in India developed through a partnership between academic researchers, a health tech company, a public health nonprofit, and a hospital. The system combines (1) stage-aware triage, routing high-risk queries to expert templates, (2) hybrid retrieval over curated maternal/newborn guidelines, and (3) evidence-conditioned generation from an LLM. Our core contribution is an evaluation workflow for high-stakes deployment under limited expert supervision. Targeting both component-level and end-to-end testing, we introduce: (i) a labeled triage benchmark (N=150) achieving 86.7% emergency recall, explicitly reporting the missed-emergency vs. over-escalation trade-off; (ii) a synthetic multi-evidence retrieval benchmark (N=100) with chunk-level evidence labels; (iii) LLM-as-judge comparison on real queries (N=781) using clinician-codesigned criteria; and (iv) expert validation. Our findings show that trustworthy medical assistants in multilingual, noisy settings require defense-in-depth design paired with multi-method evaluation, rather than any single model and evaluation method choice.

Developing and evaluating a chatbot to support maternal health care

Abstract

The ability to provide trustworthy maternal health information using phone-based chatbots can have a significant impact, particularly in low-resource settings where users have low health literacy and limited access to care. However, deploying such systems is technically challenging: user queries are short, underspecified, and code-mixed across languages, answers require regional context-specific grounding, and partial or missing symptom context makes safe routing decisions difficult. We present a chatbot for maternal health in India developed through a partnership between academic researchers, a health tech company, a public health nonprofit, and a hospital. The system combines (1) stage-aware triage, routing high-risk queries to expert templates, (2) hybrid retrieval over curated maternal/newborn guidelines, and (3) evidence-conditioned generation from an LLM. Our core contribution is an evaluation workflow for high-stakes deployment under limited expert supervision. Targeting both component-level and end-to-end testing, we introduce: (i) a labeled triage benchmark (N=150) achieving 86.7% emergency recall, explicitly reporting the missed-emergency vs. over-escalation trade-off; (ii) a synthetic multi-evidence retrieval benchmark (N=100) with chunk-level evidence labels; (iii) LLM-as-judge comparison on real queries (N=781) using clinician-codesigned criteria; and (iv) expert validation. Our findings show that trustworthy medical assistants in multilingual, noisy settings require defense-in-depth design paired with multi-method evaluation, rather than any single model and evaluation method choice.
Paper Structure (71 sections, 3 figures, 16 tables, 3 algorithms)

This paper contains 71 sections, 3 figures, 16 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Setting and problem formulation
  3. Partnership and setting.
  4. Aims:
  5. Related Work
  6. LLMs for maternal health.
  7. Evaluation criteria.
  8. RAG evaluation.
  9. System Overview
  10. Pipeline.
  11. Guardrails.
  12. Safety Through Stage-Aware Triage
  13. Stage-Aware Taxonomy and Routing Policy
  14. Implementation of triage layer
  15. Retrieval Design
  16. ...and 56 more sections

Figures (3)

  • Figure 1: Overview of the stage-aware RAG architecture. A pre-generation safety triage layer routes high-risk queries to expert-written templates, while lower-risk queries proceed through hybrid retrieval, clinical reranking, and evidence-grounded generation with guardrails.
  • Figure 2: Expert-evaluated comparison of GPT-4-Turbo and open-source models (Mixtral, LLaMA) under the RAG configuration. Scores reflect average expert ratings across medical correctness, completeness, clarity, and cultural appropriateness (lower is better).
  • Figure 3: Expert-evaluated comparison of GPT-4-Turbo and open-source models (Mixtral, LLaMA) under the No-RAG configuration. Scores reflect average expert ratings across medical correctness, completeness, clarity, and cultural appropriateness (lower is better).