A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews

TL;DR

The paper addresses the challenge of mining actionable, business-directed suggestions from unstructured customer reviews, where intent is mixed and explicit directives are rare. It proposes a hybrid pipeline that first uses a high-recall RoBERTa classifier trained with a precision--recall surrogate objective, followed by an instruction-tuned, quantized LLM (Gemma-3 27B) to extract, categorize, cluster, and summarize explicit suggestions, with local deployment considerations. The authors demonstrate through extensive experiments across restaurant and ice-cream domains that the hybrid system outperforms prompt-only, rule-based, and classifier-only baselines in extraction quality, cluster coherence, and interpretability, with strong human-evaluated usefulness. They also show that the precision--recall surrogate meaningfully improves recall (critical since missed suggestions are unrecoverable) and discuss practical limitations such as domain adaptation and deployment efficiency, offering a viable path for scalable, actionable feedback pipelines in operational settings. $L_{total} = \alpha L_{CE} + (1-\alpha) L_{PR}$ encapsulates the training objective, balancing probabilistic accuracy with recall-oriented optimization.$

Abstract

Extracting actionable suggestions from customer reviews is essential for operational decision-making, yet these directives are often embedded within mixed-intent, unstructured text. Existing approaches either classify suggestion-bearing sentences or generate high-level summaries, but rarely isolate the precise improvement instructions businesses need. We evaluate a hybrid pipeline combining a high-recall RoBERTa classifier trained with a precision-recall surrogate to reduce unrecoverable false negatives with a controlled, instruction-tuned LLM for suggestion extraction, categorization, clustering, and summarization. Across real-world hospitality and food datasets, the hybrid system outperforms prompt-only, rule-based, and classifier-only baselines in extraction accuracy and cluster coherence. Human evaluations further confirm that the resulting suggestions and summaries are clear, faithful, and interpretable. Overall, our results show that hybrid reasoning architectures achieve meaningful improvements fine-grained actionable suggestion mining while highlighting challenges in domain adaptation and efficient local deployment.

Figures (2)

• Figure 1: Overall process flow of the proposed method.
• Figure 2: Learning curve showing recall as a function of training data size. Performance saturates around 70% of the dataset, indicating that the dataset is sufficiently large for the classification task.