Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Addressing the Ecological Fallacy in Larger LMs with Human Context

Nikita Soni, Dhruv Vijay Kunjadiya, Pratham Piyush Shah, Dikshya Mohanty, H. Andrew Schwartz, Niranjan Balasubramanian

TL;DR

Empirical comparisons show that addressing the ecological fallacy during fine-tuning alone using QLoRA improves the performance of the larger 8B model over standard fine-tuning, indicating the utility and importance of modeling language in the context of its original generators, the authors.

Abstract

Language model training and inference ignore a fundamental linguistic fact -- there is a dependence between multiple sequences of text written by the same person. Prior work has shown that addressing this form of \textit{ecological fallacy} can greatly improve the performance of multiple smaller (~124M) GPT-based models. In this work, we ask if addressing the ecological fallacy by modeling the author's language context with a specific LM task (called HuLM) can provide similar benefits for a larger-scale model, an 8B Llama model. To this end, we explore variants that process an author's language in the context of their other temporally ordered texts. We study the effect of pre-training with this author context using the HuLM objective, as well as using it during fine-tuning with author context (\textit{HuFT:Human-aware Fine-Tuning}). Empirical comparisons show that addressing the ecological fallacy during fine-tuning alone using QLoRA improves the performance of the larger 8B model over standard fine-tuning. Additionally, QLoRA-based continued HuLM pre-training results in a human-aware model generalizable for improved performance over eight downstream tasks with linear task classifier training alone. These results indicate the utility and importance of modeling language in the context of its original generators, the authors.

Addressing the Ecological Fallacy in Larger LMs with Human Context

TL;DR

Empirical comparisons show that addressing the ecological fallacy during fine-tuning alone using QLoRA improves the performance of the larger 8B model over standard fine-tuning, indicating the utility and importance of modeling language in the context of its original generators, the authors.

Abstract

Language model training and inference ignore a fundamental linguistic fact -- there is a dependence between multiple sequences of text written by the same person. Prior work has shown that addressing this form of \textit{ecological fallacy} can greatly improve the performance of multiple smaller (~124M) GPT-based models. In this work, we ask if addressing the ecological fallacy by modeling the author's language context with a specific LM task (called HuLM) can provide similar benefits for a larger-scale model, an 8B Llama model. To this end, we explore variants that process an author's language in the context of their other temporally ordered texts. We study the effect of pre-training with this author context using the HuLM objective, as well as using it during fine-tuning with author context (\textit{HuFT:Human-aware Fine-Tuning}). Empirical comparisons show that addressing the ecological fallacy during fine-tuning alone using QLoRA improves the performance of the larger 8B model over standard fine-tuning. Additionally, QLoRA-based continued HuLM pre-training results in a human-aware model generalizable for improved performance over eight downstream tasks with linear task classifier training alone. These results indicate the utility and importance of modeling language in the context of its original generators, the authors.
Paper Structure (31 sections, 1 equation, 1 figure, 23 tables)

This paper contains 31 sections, 1 equation, 1 figure, 23 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Models and Methodology
  4. Processing Data within the Human Context.
  5. Classifier Training with the Human Context.
  6. Human-aware Fine-Tuning: HuFT.
  7. Continued HuLM Pre-training: HU-Llama.
  8. Baselines: Traditional Fine-tuning (TFT), Llama, LlamaLHLC.
  9. Training Setup: QLoRA Training, and Experimental Settings.
  10. Datasets and Tasks
  11. Pre-training Data: Large Human Language Corpus (LHLC)
  12. Downstream Datasets and Tasks
  13. Person-Level.
  14. Document-Level.
  15. Results and Discussion
  16. ...and 16 more sections

Figures (1)

  • Figure 1: Addressing the ecological fallacy in larger LMs with human context involving processing documents, pre-training, and fine-tuning within the author's context. Human Language Modeling (HuLM) and Human-aware Fine-Tuning (HuFT) consist of training objectives conditioned on author's historical language as the human context ($U$).