Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

I Know Therefore I Score: Label-Free Crafting of Scoring Functions using Constraints Based on Domain Expertise

Ragja Palakkadavath, Sarath Sivaprasad, Shirish Karande, Niranjan Pedanekar

TL;DR

The paper addresses the problem of crafting concise scoring functions in the absence of labeled data by introducing LFWS, a label-free weakly supervised framework that encodes domain knowledge as constraint-derived losses. The method jointly optimizes a neural scoring function with losses for monotonicity, boundedness, relative sensitivity, and output distribution, using L_f(x) = α BL + β SL + γ KL as the objective. Evaluations on a synthetic dataset and four real-world datasets (IMDB, CWUR, Journal Rank, Ad Campaign) show that LFWS achieves competitive ranking quality while ensuring the constraints are satisfied, offering a transparent and explainable alternative to fully supervised models. The work highlights practical benefits and future directions, including automatic tuning of loss weights and user-friendly validation to facilitate real-world adoption.

Abstract

Several real-life applications require crafting concise, quantitative scoring functions (also called rating systems) from measured observations. For example, an effectiveness score needs to be created for advertising campaigns using a number of engagement metrics. Experts often need to create such scoring functions in the absence of labelled data, where the scores need to reflect business insights and rules as understood by the domain experts. Without a way to capture these inputs systematically, this becomes a time-consuming process involving trial and error. In this paper, we introduce a label-free practical approach to learn a scoring function from multi-dimensional numerical data. The approach incorporates insights and business rules from domain experts in the form of easily observable and specifiable constraints, which are used as weak supervision by a machine learning model. We convert such constraints into loss functions that are optimized simultaneously while learning the scoring function. We examine the efficacy of the approach using a synthetic dataset as well as four real-life datasets, and also compare how it performs vis-a-vis supervised learning models.

I Know Therefore I Score: Label-Free Crafting of Scoring Functions using Constraints Based on Domain Expertise

TL;DR

The paper addresses the problem of crafting concise scoring functions in the absence of labeled data by introducing LFWS, a label-free weakly supervised framework that encodes domain knowledge as constraint-derived losses. The method jointly optimizes a neural scoring function with losses for monotonicity, boundedness, relative sensitivity, and output distribution, using L_f(x) = α BL + β SL + γ KL as the objective. Evaluations on a synthetic dataset and four real-world datasets (IMDB, CWUR, Journal Rank, Ad Campaign) show that LFWS achieves competitive ranking quality while ensuring the constraints are satisfied, offering a transparent and explainable alternative to fully supervised models. The work highlights practical benefits and future directions, including automatic tuning of loss weights and user-friendly validation to facilitate real-world adoption.

Abstract

Several real-life applications require crafting concise, quantitative scoring functions (also called rating systems) from measured observations. For example, an effectiveness score needs to be created for advertising campaigns using a number of engagement metrics. Experts often need to create such scoring functions in the absence of labelled data, where the scores need to reflect business insights and rules as understood by the domain experts. Without a way to capture these inputs systematically, this becomes a time-consuming process involving trial and error. In this paper, we introduce a label-free practical approach to learn a scoring function from multi-dimensional numerical data. The approach incorporates insights and business rules from domain experts in the form of easily observable and specifiable constraints, which are used as weak supervision by a machine learning model. We convert such constraints into loss functions that are optimized simultaneously while learning the scoring function. We examine the efficacy of the approach using a synthetic dataset as well as four real-life datasets, and also compare how it performs vis-a-vis supervised learning models.
Paper Structure (24 sections, 6 equations, 2 figures, 3 tables)

This paper contains 24 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Crafting Scoring Functions
  4. Monotonicity Constraint
  5. Boundedness Constraint
  6. Sensitivity Constraint
  7. Output Constraint
  8. Gaussian Distribution
  9. Exponential Distribution
  10. Experimental Details
  11. Datasets
  12. Synthetic Dataset
  13. IMDB Dataset
  14. CWUR Dataset
  15. Journal Rank Dataset
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Kernel density estimates of generated scores
  • Figure 2: Kernel density estimates for generated scores