Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Maximizing mutual information between user-contexts and responses improve LLM personalization with no additional data

Hyunji Nam, Haoran Li, Natasha Jaques

Abstract

While post-training has successfully improved large language models (LLMs) across a variety of domains, these gains heavily rely on human-labeled data or external verifiers. Existing data has already been exploited, and new high-quality data is expensive to collect. More fundamentally, true intelligence goes far beyond tasks that are easily verifiable. Therefore, we need self-improvement frameworks that allow models to improve without external oversight. We propose *Mutual Information Preference Optimization (MIPO)*, a contrastive data augmentation method that constructs preference pairs by generating a positive response conditioning on the correct prompt, and a negative response by conditioning on a random, unrelated prompt. We show that using Direct Preference Optimization (DPO) to learn from this paired data maximizes pointwise conditional mutual information (MI) (under the base LLM) between prompts and model responses. Empirical results with various-sized Llama- and Qwen-Instruct models show that when used to maximize MI between user context and response, MIPO provides an effective personalization technique, achieving 3-40% improvements on personalization tasks using real-user datasets compared to strong baselines. Surprisingly, MIPO can also be applied to improve performance on math and multiple-choice problems, yielding 1-18% **without any additional data or human supervision**. These results suggest a promising direction for self-improvement.

Maximizing mutual information between user-contexts and responses improve LLM personalization with no additional data

Abstract

While post-training has successfully improved large language models (LLMs) across a variety of domains, these gains heavily rely on human-labeled data or external verifiers. Existing data has already been exploited, and new high-quality data is expensive to collect. More fundamentally, true intelligence goes far beyond tasks that are easily verifiable. Therefore, we need self-improvement frameworks that allow models to improve without external oversight. We propose *Mutual Information Preference Optimization (MIPO)*, a contrastive data augmentation method that constructs preference pairs by generating a positive response conditioning on the correct prompt, and a negative response by conditioning on a random, unrelated prompt. We show that using Direct Preference Optimization (DPO) to learn from this paired data maximizes pointwise conditional mutual information (MI) (under the base LLM) between prompts and model responses. Empirical results with various-sized Llama- and Qwen-Instruct models show that when used to maximize MI between user context and response, MIPO provides an effective personalization technique, achieving 3-40% improvements on personalization tasks using real-user datasets compared to strong baselines. Surprisingly, MIPO can also be applied to improve performance on math and multiple-choice problems, yielding 1-18% **without any additional data or human supervision**. These results suggest a promising direction for self-improvement.
Paper Structure (19 sections, 15 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 19 sections, 15 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Mutual Information Preference Optimization (MIPO)
  5. MIPO for personalization: Maximizing conditional mutual information given prompts
  6. Experiments
  7. Baselines
  8. Mutual information reward baselines
  9. Domains
  10. Evaluation
  11. Results
  12. Limitation & Discussion
  13. Appendix: Results
  14. Appendix: MIPO-Personalized Responses
  15. Appendix: Datasets
  16. ...and 4 more sections

Figures (3)

  • Figure 1: We propose an intrinsically motivated post-training method based on mutual information that does not require human labels or verifiers. We present two versions of MIPO: (1) maximizing mutual information between responses and prompts, and (2) maximizing conditional mutual information between responses and user contexts given prompts The second objective is particularly useful for personalization, as it encourages the policy to generate responses that are more likely conditioned on the specific user context, but rare globally.
  • Figure 2: Entropy over the MCQ answer choices conditioned on correct model predictions. In addition to improving accuracy, MIPO (blue) also makes models become more confident about correct predictions compared to the base model (orange) as indicated by the mean and overall distribution shift. The x-axis ranges from mean $\pm$ 1 std.
  • Figure 3: Entropy over the 500 MCQ answer choices in ARC conditioned on output correctness. MIPO-trained models (blue) become more confident about correct answers compared to the base models (orange) as indicated by entropy reduction. The range is truncated to show the mean $\pm$ 2 standard deviation.