Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization

Aseem Srivastava, Smriti Joshi, Tanmoy Chakraborty, Md Shad Akhtar

TL;DR

This work employs a planning engine on Llama-2, resulting in a novel framework, PIECE, which employs knowledge filtering-cum-scaffolding to encapsulate domain knowledge and leverages sheaf convolution learning to enhance the dialogue’s structural nuances.

Abstract

In mental health counseling, condensing dialogues into concise and relevant summaries (aka counseling notes) holds pivotal significance. Large Language Models (LLMs) exhibit remarkable capabilities in various generative tasks; however, their adaptation to domain-specific intricacies remains challenging, especially within mental health contexts. Unlike standard LLMs, mental health experts first plan to apply domain knowledge in writing summaries. Our work enhances LLMs' ability by introducing a novel planning engine to orchestrate structuring knowledge alignment. To achieve high-order planning, we divide knowledge encapsulation into two major phases: (i) holding dialogue structure and (ii) incorporating domain-specific knowledge. We employ a planning engine on Llama-2, resulting in a novel framework, PIECE. Our proposed system employs knowledge filtering-cum-scaffolding to encapsulate domain knowledge. Additionally, PIECE leverages sheaf convolution learning to enhance its understanding of the dialogue's structural nuances. We compare PIECE with 14 baseline methods and observe a significant improvement across ROUGE and Bleurt scores. Further, expert evaluation and analyses validate the generation quality to be effective, sometimes even surpassing the gold standard. We further benchmark PIECE with other LLMs and report improvement, including Llama-2 (+2.72%), Mistral (+2.04%), and Zephyr (+1.59%), to justify the generalizability of the planning engine.

Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization

TL;DR

This work employs a planning engine on Llama-2, resulting in a novel framework, PIECE, which employs knowledge filtering-cum-scaffolding to encapsulate domain knowledge and leverages sheaf convolution learning to enhance the dialogue’s structural nuances.

Abstract

In mental health counseling, condensing dialogues into concise and relevant summaries (aka counseling notes) holds pivotal significance. Large Language Models (LLMs) exhibit remarkable capabilities in various generative tasks; however, their adaptation to domain-specific intricacies remains challenging, especially within mental health contexts. Unlike standard LLMs, mental health experts first plan to apply domain knowledge in writing summaries. Our work enhances LLMs' ability by introducing a novel planning engine to orchestrate structuring knowledge alignment. To achieve high-order planning, we divide knowledge encapsulation into two major phases: (i) holding dialogue structure and (ii) incorporating domain-specific knowledge. We employ a planning engine on Llama-2, resulting in a novel framework, PIECE. Our proposed system employs knowledge filtering-cum-scaffolding to encapsulate domain knowledge. Additionally, PIECE leverages sheaf convolution learning to enhance its understanding of the dialogue's structural nuances. We compare PIECE with 14 baseline methods and observe a significant improvement across ROUGE and Bleurt scores. Further, expert evaluation and analyses validate the generation quality to be effective, sometimes even surpassing the gold standard. We further benchmark PIECE with other LLMs and report improvement, including Llama-2 (+2.72%), Mistral (+2.04%), and Zephyr (+1.59%), to justify the generalizability of the planning engine.
Paper Structure (41 sections, 3 equations, 3 figures, 7 tables)

This paper contains 41 sections, 3 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative AI in Mental Health Domain.
  4. Knowledge Enhancement in LLMs.
  5. Text Summarization.
  6. Dataset
  7. Proposed Methodology
  8. Domain Knowledge Encapsulation
  9. Knowledge Filtering.
  10. Knowledge Scaffolding.
  11. Structural Knowledge Encapsulation
  12. Sheaf Learner.
  13. Planning Engine
  14. Experiments and Results
  15. Baselines
  16. ...and 26 more sections

Figures (3)

  • Figure 1: The proposed pipeline allows LLMs to first plan and then generate. In our approach, prioritizing planning before generation enriches summarization with conversational structure and domain knowledge.
  • Figure 2: Architecture of PIECE. We propose a novel planning engine consisting of two primary sections: (a) integrating knowledge filtering-cum-scaffolding and (b) encapsulating structural understanding of dialogues. The filtration of relevant utterances utilizes counseling component labels within the MEMO dataset to mask filler utterances, followed by knowledge scaffolding. Additionally, sheaf learners are employed for the structural understanding of counseling dialogue. The planning engine operates using a rotating attention mechanism using knowledge from both segments for better LLM generation.
  • Figure 3: Domain-centric evaluation using Mental Health Info Capture (MHIC) metric. The proposed model, PIECE, distinctly excels in capturing domain knowledge compared to the two most relevant models.