Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhanced Urdu Intent Detection with Large Language Models and Prototype-Informed Predictive Pipelines

Faiza Hassan, Summra Saleem, Kashif Javed, Muhammad Nabeel Asim, Abdur Rehman, Andreas Dengel

TL;DR

This paper tackles Urdu intent detection in few-shot settings by introducing LLMPIA, a two-stage framework that combines contrastive-learning enhanced representations (LLMRCL) with a prototype-informed attention (PIA) predictor. Evaluated on Urdu ATIS translations and Urdu Web Queries with six pre-trained language models and thirteen similarity metrics, the approach demonstrates substantial performance gains over baselines, notably with re-training and in 5-shot scenarios. The study finds that multilingual PLMs often outperform Urdu-specific models under few-shot conditions and identifies cosine similarity as a robust choice for prototype matching. Overall, LLMPIA advances Urdu NLP by enabling effective unseen-intent detection and provides actionable insights for model selection and similarity measures in low-resource language settings.

Abstract

Multifarious intent detection predictors are developed for different languages, including English, Chinese and French, however, the field remains underdeveloped for Urdu, the 10th most spoken language. In the realm of well-known languages, intent detection predictors utilize the strategy of few-shot learning and prediction of unseen classes based on the model training on seen classes. However, Urdu language lacks few-shot strategy based intent detection predictors and traditional predictors are focused on prediction of the same classes which models have seen in the train set. To empower Urdu language specific intent detection, this introduces a unique contrastive learning approach that leverages unlabeled Urdu data to re-train pre-trained language models. This re-training empowers LLMs representation learning for the downstream intent detection task. Finally, it reaps the combined potential of pre-trained LLMs and the prototype-informed attention mechanism to create a comprehensive end-to-end LLMPIA intent detection pipeline. Under the paradigm of proposed predictive pipeline, it explores the potential of 6 distinct language models and 13 distinct similarity computation methods. The proposed framework is evaluated on 2 public benchmark datasets, namely ATIS encompassing 5836 samples and Web Queries having 8519 samples. Across ATIS dataset under 4-way 1 shot and 4-way 5 shot experimental settings LLMPIA achieved 83.28% and 98.25% F1-Score and on Web Queries dataset produced 76.23% and 84.42% F1-Score, respectively. In an additional case study on the Web Queries dataset under same classes train and test set settings, LLMPIA outperformed state-of-the-art predictor by 53.55% F1-Score.

Enhanced Urdu Intent Detection with Large Language Models and Prototype-Informed Predictive Pipelines

TL;DR

This paper tackles Urdu intent detection in few-shot settings by introducing LLMPIA, a two-stage framework that combines contrastive-learning enhanced representations (LLMRCL) with a prototype-informed attention (PIA) predictor. Evaluated on Urdu ATIS translations and Urdu Web Queries with six pre-trained language models and thirteen similarity metrics, the approach demonstrates substantial performance gains over baselines, notably with re-training and in 5-shot scenarios. The study finds that multilingual PLMs often outperform Urdu-specific models under few-shot conditions and identifies cosine similarity as a robust choice for prototype matching. Overall, LLMPIA advances Urdu NLP by enabling effective unseen-intent detection and provides actionable insights for model selection and similarity measures in low-resource language settings.

Abstract

Multifarious intent detection predictors are developed for different languages, including English, Chinese and French, however, the field remains underdeveloped for Urdu, the 10th most spoken language. In the realm of well-known languages, intent detection predictors utilize the strategy of few-shot learning and prediction of unseen classes based on the model training on seen classes. However, Urdu language lacks few-shot strategy based intent detection predictors and traditional predictors are focused on prediction of the same classes which models have seen in the train set. To empower Urdu language specific intent detection, this introduces a unique contrastive learning approach that leverages unlabeled Urdu data to re-train pre-trained language models. This re-training empowers LLMs representation learning for the downstream intent detection task. Finally, it reaps the combined potential of pre-trained LLMs and the prototype-informed attention mechanism to create a comprehensive end-to-end LLMPIA intent detection pipeline. Under the paradigm of proposed predictive pipeline, it explores the potential of 6 distinct language models and 13 distinct similarity computation methods. The proposed framework is evaluated on 2 public benchmark datasets, namely ATIS encompassing 5836 samples and Web Queries having 8519 samples. Across ATIS dataset under 4-way 1 shot and 4-way 5 shot experimental settings LLMPIA achieved 83.28% and 98.25% F1-Score and on Web Queries dataset produced 76.23% and 84.42% F1-Score, respectively. In an additional case study on the Web Queries dataset under same classes train and test set settings, LLMPIA outperformed state-of-the-art predictor by 53.55% F1-Score.
Paper Structure (18 sections, 14 equations, 5 figures, 8 tables)

This paper contains 18 sections, 14 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Research Objective
  4. LLMPIA-Training Phase
  5. LLMPIA-Validation and LLMPIA-Testing Phases
  6. Proposed Framework
  7. LLMRCL: Large Language Models Enhanced Representations with Contrastive Learning based re-training
  8. Prototype Informed Attention Model
  9. Training Process
  10. Datasets and Evaluation Measures
  11. Datasets
  12. Evaluation Measures
  13. Experimental Setup and Results
  14. Language Models re-training impact on Predictive Pipelines Performance
  15. Proposed framework predictive pipelines robustness analysis
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Graphical illustration of Key Steps of Large Language Models Enhanced Representations with Contrastive Learning based re-training: 1) Masked Language Modelling, 2) Self-Supervised Contrastive Learning
  • Figure 2: Graphical illustration of the key steps in the Prototype Informed Attention Approach for Urdu intent detection
  • Figure 3: Graphical overview of dataset statistics for Urdu intent detection
  • Figure 4: A Comprehensive Performance Analysis of Distinct Predictive Pipelines of Proposed Framework in terms of Precision and Recall over ATIS Dataset
  • Figure 5: A Comprehensive Performance Analysis of Distinct Predictive Pipelines of Proposed Framework In terms of Precision and Recall over Web Queries dataset