Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IndoBERT-Relevancy: A Context-Conditioned Relevancy Classifier for Indonesian Text

Muhammad Apriandito Arya Saputra, Andry Alamsyah, Dian Puteri Ramadhani, Thomhert Suprapto Siadari, Hanif Fakhrurroja

Abstract

Determining whether a piece of text is relevant to a given topic is a fundamental task in natural language processing, yet it remains largely unexplored for Bahasa Indonesia. Unlike sentiment analysis or named entity recognition, relevancy classification requires the model to reason about the relationship between two inputs simultaneously: a topical context and a candidate text. We introduce IndoBERT-Relevancy, a context-conditioned relevancy classifier built on IndoBERT Large (335M parameters) and trained on a novel dataset of 31,360 labeled pairs spanning 188 topics. Through an iterative, failure-driven data construction process, we demonstrate that no single data source is sufficient for robust relevancy classification, and that targeted synthetic data can effectively address specific model weaknesses. Our final model achieves an F1 score of 0.948 and an accuracy of 96.5%, handling both formal and informal Indonesian text. The model is publicly available at HuggingFace.

IndoBERT-Relevancy: A Context-Conditioned Relevancy Classifier for Indonesian Text

Abstract

Determining whether a piece of text is relevant to a given topic is a fundamental task in natural language processing, yet it remains largely unexplored for Bahasa Indonesia. Unlike sentiment analysis or named entity recognition, relevancy classification requires the model to reason about the relationship between two inputs simultaneously: a topical context and a candidate text. We introduce IndoBERT-Relevancy, a context-conditioned relevancy classifier built on IndoBERT Large (335M parameters) and trained on a novel dataset of 31,360 labeled pairs spanning 188 topics. Through an iterative, failure-driven data construction process, we demonstrate that no single data source is sufficient for robust relevancy classification, and that targeted synthetic data can effectively address specific model weaknesses. Our final model achieves an F1 score of 0.948 and an accuracy of 96.5%, handling both formal and informal Indonesian text. The model is publicly available at HuggingFace.

Paper Structure

This paper contains 25 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Task Definition
  3. Why This Is Hard for Indonesian
  4. The Register Spectrum
  5. Implicit Expression
  6. Building the Dataset
  7. Stage 1: Formal Text Pairs
  8. Stage 2: Informal Text Pairs
  9. Stage 3: Implicit Text Generation
  10. Model
  11. Architecture
  12. Handling Class Imbalance
  13. Training
  14. Results
  15. Progressive Improvement
  16. ...and 10 more sections

Figures (3)

  • Figure 1: The iterative, failure-driven data construction process. Each stage is motivated by a specific weakness identified in the previous model. Dashed red arrows indicate the identified failure that motivated the next stage.
  • Figure 2: Model performance improves with each data source. The largest single improvement comes from adding informal text (+6.2 F1), while implicit text provides a targeted boost (+2.6 F1).
  • Figure 3: Validation metrics across training epochs for the final model (V3).