Human interaction classifier for LLM based chatbot
Diego Martín, Jordi Sanchez, Xavier Vizcaíno
TL;DR
The paper tackles the problem of routing user interactions in an AI-powered assistant by classifying inputs into Conversation, Services, and Document Translation. It systematically compares LLM-based classifiers, KNN with Titan and Cohere embeddings, and classical ML approaches (SVM and ANN) using embedding features. The study finds that SVM and ANN models with Cohere embeddings deliver the best overall performance, with SVM achieving F1 scores of 0.99 for Conversation, 0.80 for Services, and 0.93 for Document Translation, while maintaining fast inference times; LLM-based approaches lag in accuracy and efficiency, though providing examples can boost some tasks. These results provide actionable guidance for deploying efficient, high-precision interaction classifiers in multilingual, production-ready AI assistants like AIDA, enabling effective routing to specialized pipelines and improved user experience.
Abstract
This study investigates different approaches to classify human interactions in an artificial intelligence-based environment, specifically for Applus+ IDIADA's intelligent agent AIDA. The main objective is to develop a classifier that accurately identifies the type of interaction received (Conversation, Services, or Document Translation) to direct requests to the appropriate channel and provide a more specialized and efficient service. Various models are compared, including LLM-based classifiers, KNN using Titan and Cohere embeddings, SVM, and artificial neural networks. Results show that SVM and ANN models with Cohere embeddings achieve the best overall performance, with superior F1 scores and faster execution times compared to LLM-based approaches. The study concludes that the SVM model with Cohere embeddings is the most suitable option for classifying human interactions in the AIDA environment, offering an optimal balance between accuracy and computational efficiency.