PeerCoPilot: A Language Model-Powered Assistant for Behavioral Health Organizations

TL;DR

PeerCoPilot introduces a retrieval-augmented, LLM-powered assistant to support peer providers in PROs by generating wellness plans, SMART goals, and location-specific resources. The system integrates a vetted resource database with modular backend components to ensure reliability and holistic guidance across the eight_dimensions of wellness. Human evaluations (with 15 peer providers and 6 service users) show high willingness to adopt PeerCoPilot and a clear edge in reliability and specificity over a baseline LLM, with preliminary teaming studies indicating faster, higher-quality wellness plans. Deployed at CSPNJ and in expansion, PeerCoPilot demonstrates practical potential for augmenting peer support while addressing staffing constraints in behavioral health services.

Abstract

Behavioral health conditions, which include mental health and substance use disorders, are the leading disease burden in the United States. Peer-run behavioral health organizations (PROs) critically assist individuals facing these conditions by combining mental health services with assistance for needs such as income, employment, and housing. However, limited funds and staffing make it difficult for PROs to address all service user needs. To assist peer providers at PROs with their day-to-day tasks, we introduce PeerCoPilot, a large language model (LLM)-powered assistant that helps peer providers create wellness plans, construct step-by-step goals, and locate organizational resources to support these goals. PeerCoPilot ensures information reliability through a retrieval-augmented generation pipeline backed by a large database of over 1,300 vetted resources. We conducted human evaluations with 15 peer providers and 6 service users and found that over 90% of users supported using PeerCoPilot. Moreover, we demonstrated that PeerCoPilot provides more reliable and specific information than a baseline LLM. PeerCoPilot is now used by a group of 5-10 peer providers at CSPNJ, a large behavioral health organization serving over 10,000 service users, and we are actively expanding PeerCoPilot's use.

Figures (9)

• Figure 1: PeerCoPilot assists peer providers at pros during peer sessions with service users. Because information reliability is critical, PeerCoPilot combines LLMs with trusted sources to ensure accuracy.
• Figure 2: PeerCoPilot takes in a peer provider's input and passes this into four modules: resource recommendation, benefit eligibility, goal construction, and question generation. Some modules combine the peer provider's query with externally verified information to ensure accuracy. The results from all modules are combined to form a response that tackles various aspects, including goals, resources, and follow-up questions.
• Figure 3: PeerCoPilot is a chat-based frontend that presents peer providers with goals, resources, and follow-up questions pertinent to a service user's situation. Peer providers can also ask further questions to PeerCoPilot.
• Figure 4: We surveyed peer providers on the usability of PeerCoPilot. All peer providers found PeerCoPilot simple and are willing to use it in practice.
• Figure 5: Peer providers found that PeerCoPilot generates better questions, recommends better resources, and crafts better goals compared to a baseline. This is because the modules ensure information reliability and specificity.