HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance

TL;DR

HuixiangDou presents a group-chat oriented technical assistant that uses a three-stage pipeline (Preprocess, Rejection, Response) to deliver grounded, domain-specific answers from OpenMMLab projects while minimizing noise and hallucinations. The approach combines a text2vec rejection filter with LLM-based scoring, keyword extraction, web search, and a knowledge-graph–driven retrieval system, orchestrated via a hybrid LLM service to balance cost and capability. Key contributions include a rejection mechanism validated by text2vec precision/recall, an LLM scoring framework for intent and relevance, and long-context optimization enabling up to tens of thousands of tokens, all demonstrated through fine-tuning experiments and ablations. The work demonstrates practical viability for deployment in real group chats and highlights future directions for data quality, multimodal inputs, and enhanced pretraining to further close the gap between in-chat questions and accurate, domain-specific responses.

Abstract

In this work, we present HuixiangDou, a technical assistant powered by Large Language Models (LLM). This system is designed to assist algorithm developers by providing insightful responses to questions related to open-source algorithm projects, such as computer vision and deep learning projects from OpenMMLab. We further explore the integration of this assistant into the group chats of instant messaging (IM) tools such as WeChat and Lark. Through several iterative improvements and trials, we have developed a sophisticated technical chat assistant capable of effectively answering users' technical questions without causing message flooding. This paper's contributions include: 1) Designing an algorithm pipeline specifically for group chat scenarios; 2) Verifying the reliable performance of text2vec in task rejection; 3) Identifying three critical requirements for LLMs in technical-assistant-like products, namely scoring ability, In-Context Learning (ICL), and Long Context. We have made the source code, android app and web service available at Github (https://github.com/internlm/huixiangdou), OpenXLab (https://openxlab.org.cn/apps/detail/tpoisonooo/huixiangdou-web) and YouTube (https://youtu.be/ylXrT-Tei-Y) to aid in future research and application. HuixiangDou is applicable to any group chat within IM tools.

Figures (9)

• Figure 1: The overall structure of the approach. After the user's message is preprocessed, small talk will be filtered out, and only genuine questions will be responded to.
• Figure 2: The structure of rejection pipeline. We build a two-stage refusal-to-answer filter using text2vec and LLM scoring.
• Figure 3: The structure of response pipeline. We employ scoring and partial ordering to filter high-quality text from rerank model, web search and knowledge graph for the LLM to generate responses. To save costs, we mix and schedule different LLMs. We have established a set of security mechanisms to ensure that replies to chat groups do not involve sensitive topics.
• Figure 4: Question likelihood with InternLM2-7B on 1,303 domain-related group chat sentences, 11.6% are questions. This distribution aligns with common sense, and the scoring method can effectively handle intent recognition tasks.
• Figure 5: RaR prompt is not applicable to interrogative sentences.