Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Knowledge Distillation on Text Graph for Data-limited Attribute Inference

Quan Li, Shixiong Jing, Lingwei Chen

TL;DR

This work tackles user attribute inference from social media text under severe label scarcity. It builds a text graph over texts using manifold learning and refines it with task-driven message passing, instead of a word-centric graph, to improve few-shot Propagation. It further introduces hierarchical knowledge distillation to leverage cross-domain labeled/unlabeled data and target-domain unlabeled data, achieving robust generalization in transductive training. Empirical results on GeoText, Twitter, and Blog datasets show state-of-the-art performance in few-shot scenarios and underscore the importance of graph refinement and two-level distillation for practical, data-limited attribute inference. The approach offers a scalable, effective framework for privacy-sensitive attribute inference tasks in social media analysis.

Abstract

The popularization of social media increases user engagements and generates a large amount of user-oriented data. Among them, text data (e.g., tweets, blogs) significantly attracts researchers and speculators to infer user attributes (e.g., age, gender, location) for fulfilling their intents. Generally, this line of work casts attribute inference as a text classification problem, and starts to leverage graph neural networks (GNNs) to utilize higher-level representations of source texts. However, these text graphs are constructed over words, suffering from high memory consumption and ineffectiveness on few labeled texts. To address this challenge, we design a text-graph-based few-shot learning model for attribute inferences on social media text data. Our model first constructs and refines a text graph using manifold learning and message passing, which offers a better trade-off between expressiveness and complexity. Afterwards, to further use cross-domain texts and unlabeled texts to improve few-shot performance, a hierarchical knowledge distillation is devised over text graph to optimize the problem, which derives better text representations, and advances model generalization ability. Experiments on social media datasets demonstrate the state-of-the-art performance of our model on attribute inferences with considerably fewer labeled texts.

Hierarchical Knowledge Distillation on Text Graph for Data-limited Attribute Inference

TL;DR

This work tackles user attribute inference from social media text under severe label scarcity. It builds a text graph over texts using manifold learning and refines it with task-driven message passing, instead of a word-centric graph, to improve few-shot Propagation. It further introduces hierarchical knowledge distillation to leverage cross-domain labeled/unlabeled data and target-domain unlabeled data, achieving robust generalization in transductive training. Empirical results on GeoText, Twitter, and Blog datasets show state-of-the-art performance in few-shot scenarios and underscore the importance of graph refinement and two-level distillation for practical, data-limited attribute inference. The approach offers a scalable, effective framework for privacy-sensitive attribute inference tasks in social media analysis.

Abstract

The popularization of social media increases user engagements and generates a large amount of user-oriented data. Among them, text data (e.g., tweets, blogs) significantly attracts researchers and speculators to infer user attributes (e.g., age, gender, location) for fulfilling their intents. Generally, this line of work casts attribute inference as a text classification problem, and starts to leverage graph neural networks (GNNs) to utilize higher-level representations of source texts. However, these text graphs are constructed over words, suffering from high memory consumption and ineffectiveness on few labeled texts. To address this challenge, we design a text-graph-based few-shot learning model for attribute inferences on social media text data. Our model first constructs and refines a text graph using manifold learning and message passing, which offers a better trade-off between expressiveness and complexity. Afterwards, to further use cross-domain texts and unlabeled texts to improve few-shot performance, a hierarchical knowledge distillation is devised over text graph to optimize the problem, which derives better text representations, and advances model generalization ability. Experiments on social media datasets demonstrate the state-of-the-art performance of our model on attribute inferences with considerably fewer labeled texts.
Paper Structure (13 sections, 14 equations, 2 figures, 3 tables, 1 algorithm)

This paper contains 13 sections, 14 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Proposed Model
  4. Text Representations
  5. Text Graph Construction and Refinement
  6. Hierarchical Knowledge Distillation
  7. Loss Generation for Transductive Training
  8. Experimental Results and Analysis
  9. Experimental Setup
  10. Evaluation of Our Model
  11. Comparisons with Baselines
  12. Ablation Study
  13. Conclusion

Figures (2)

  • Figure 1: The overview of our proposed model, which includes three main components: text representations, text graph construction and refinement, and hierarchical knowledge distillation.
  • Figure 2: Evaluation on different model parameters: (a) sizes of training samples $m$, (b) distillation temperatures $\tau$, and (c) distillation balance parameter $\lambda$.