Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AgriLens: Semantic Retrieval in Agricultural Texts Using Topic Modeling and Language Models

Heba Shakeel, Tanvir Ahmad, Tanya Liyaqat, Chandni Saxena

TL;DR

AgriLens addresses the challenge of scalable, interpretable retrieval from large, unstructured agricultural text where labeled data are scarce. It combines BERTopic-based fine-grained topic modeling with zero-shot topic labeling via language models and topic-guided semantic retrieval over dense embeddings, enabling vector-search based exploration of agricultural narratives. The approach is evaluated on ENAGRINEWS, demonstrating coherent topic discovery, meaningful label generation, and effective retrieval, supplemented by bias and grounding assessments and human evaluations. The framework aims to provide accessible, domain-specific information access for researchers, policymakers, and practitioners, with reproducible pipelines and a public codebase.

Abstract

As the volume of unstructured text continues to grow across domains, there is an urgent need for scalable methods that enable interpretable organization, summarization, and retrieval of information. This work presents a unified framework for interpretable topic modeling, zero-shot topic labeling, and topic-guided semantic retrieval over large agricultural text corpora. Leveraging BERTopic, we extract semantically coherent topics. Each topic is converted into a structured prompt, enabling a language model to generate meaningful topic labels and summaries in a zero-shot manner. Querying and document exploration are supported via dense embeddings and vector search, while a dedicated evaluation module assesses topical coherence and bias. This framework supports scalable and interpretable information access in specialized domains where labeled data is limited.

AgriLens: Semantic Retrieval in Agricultural Texts Using Topic Modeling and Language Models

TL;DR

AgriLens addresses the challenge of scalable, interpretable retrieval from large, unstructured agricultural text where labeled data are scarce. It combines BERTopic-based fine-grained topic modeling with zero-shot topic labeling via language models and topic-guided semantic retrieval over dense embeddings, enabling vector-search based exploration of agricultural narratives. The approach is evaluated on ENAGRINEWS, demonstrating coherent topic discovery, meaningful label generation, and effective retrieval, supplemented by bias and grounding assessments and human evaluations. The framework aims to provide accessible, domain-specific information access for researchers, policymakers, and practitioners, with reproducible pipelines and a public codebase.

Abstract

As the volume of unstructured text continues to grow across domains, there is an urgent need for scalable methods that enable interpretable organization, summarization, and retrieval of information. This work presents a unified framework for interpretable topic modeling, zero-shot topic labeling, and topic-guided semantic retrieval over large agricultural text corpora. Leveraging BERTopic, we extract semantically coherent topics. Each topic is converted into a structured prompt, enabling a language model to generate meaningful topic labels and summaries in a zero-shot manner. Querying and document exploration are supported via dense embeddings and vector search, while a dedicated evaluation module assesses topical coherence and bias. This framework supports scalable and interpretable information access in specialized domains where labeled data is limited.
Paper Structure (19 sections, 5 equations, 5 figures, 5 tables)

This paper contains 19 sections, 5 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Framework and System Architecture
  4. Data and Pre-processing
  5. Corpus Description
  6. Text Extraction and Data Cleaning
  7. Document Chunking
  8. Topic Modeling and Label Generation
  9. Fine-Grained Topic Modeling
  10. Topic Label Generation using Language Models
  11. Language models configuration
  12. Topic Labeling
  13. Experimental setup
  14. Evaluation of Generated Labels
  15. Semantic Retrieval and Evaluation
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Overview of the AgriLens system: Representation of the topic-level indexing framework. A semantic "Lens" scans a large corpus of unstructured documents to extract and assign concise topic labels.
  • Figure 2: Proposed pipeline for end-to-end semantic search
  • Figure 3: Example snippet from ENARGINEWS document
  • Figure 4: Workflow of topic label generation.
  • Figure 5: A snapshot of AgriLens system usage.