Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can LLMs Simulate Personas with Reversed Performance? A Systematic Investigation for Counterfactual Instruction Following in Math Reasoning Context

Sai Adith Senthil Kumar, Hao Yan, Saipavan Perepa, Murong Yue, Ziyu Yao

Abstract

Large Language Models (LLMs) are now increasingly widely used to simulate personas in virtual environments, leveraging their instruction-following capability. However, we discovered that even state-of-the-art LLMs cannot simulate personas with reversed performance (e.g., student personas with low proficiency in educational settings), which impairs the simulation diversity and limits the practical applications of the simulated environments. In this work, using mathematical reasoning as a representative scenario, we propose the first benchmark dataset for evaluating LLMs on simulating personas with reversed performance, a capability that we dub "counterfactual instruction following". We evaluate both open-weight and closed-source LLMs on this task and find that LLMs, including the OpenAI o1 reasoning model, all struggle to follow counterfactual instructions for simulating reversedly performing personas. Intersectionally simulating both the performance level and the race population of a persona worsens the effect even further. These results highlight the challenges of counterfactual instruction following and the need for further research.

Can LLMs Simulate Personas with Reversed Performance? A Systematic Investigation for Counterfactual Instruction Following in Math Reasoning Context

Abstract

Large Language Models (LLMs) are now increasingly widely used to simulate personas in virtual environments, leveraging their instruction-following capability. However, we discovered that even state-of-the-art LLMs cannot simulate personas with reversed performance (e.g., student personas with low proficiency in educational settings), which impairs the simulation diversity and limits the practical applications of the simulated environments. In this work, using mathematical reasoning as a representative scenario, we propose the first benchmark dataset for evaluating LLMs on simulating personas with reversed performance, a capability that we dub "counterfactual instruction following". We evaluate both open-weight and closed-source LLMs on this task and find that LLMs, including the OpenAI o1 reasoning model, all struggle to follow counterfactual instructions for simulating reversedly performing personas. Intersectionally simulating both the performance level and the race population of a persona worsens the effect even further. These results highlight the challenges of counterfactual instruction following and the need for further research.

Paper Structure

This paper contains 25 sections, 9 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Reversed Persona Simulation as Counterfactual Instruction Following
  3. Task Setting
  4. Evaluation Metrics
  5. Methodologies
  6. Experiment
  7. Experimental Setting
  8. Can LLMs simulate personas with reversed performance by following counterfactual instructions?
  9. How does intersectional persona simulation affect LLMs in following counterfactual instructions?
  10. Additional Analysis
  11. Related Work
  12. LLM Counterfactual Instruction Following
  13. LLMs for Persona Simulation
  14. Conclusion
  15. Additional Task Settings
  16. ...and 10 more sections

Figures (9)

  • Figure 1: Illustration of evaluating LLMs for simulating personas with high- and low-proficiency in math reasoning. For low-performing persona simulation, we additionally instruct the model to leave comments and explain evidence of low performance, which encourages more faithful simulation and helps response analysis.
  • Figure 2: Prompt templates for different prompting approaches. [] denotes a placeholder for specified content.
  • Figure 3: Changes in task accuracy for high- and low-performing persona simulations when race attributes are added in zero-shot prompting, relative to the setting without race.
  • Figure 4: Instructions for race population.
  • Figure 5: Prompt templates for different prompting approaches used in intersectional simulation. [] denotes a placeholder for specified content.
  • ...and 4 more figures