Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mimetic Alignment with ASPECT: Evaluation of AI-inferred Personal Profiles

Ruoxi Shang, Dan Marshall, Edward Cutrell, Denae Ford

Abstract

AI agents that communicate on behalf of individuals need to capture how each person actually communicates, yet current approaches either require costly per-person fine-tuning, produce generic outputs from shallow persona descriptions, or optimize preferences without modeling communication style. We present ASPECT (Automated Social Psychometric Evaluation of Communication Traits), a pipeline that directs LLMs to assess constructs from a validated communication scale against behavioral evidence from workplace data, without per-person training. In a case study with 20 participants (1,840 paired item ratings, 600 scenario evaluations), ASPECT-generated profiles achieved moderate alignment with self-assessments, and ASPECT-generated responses were preferred over generic and self-report baselines on aggregate, with substantial variation across individuals and scenarios. During the profile review phase, linked evidence helped participants identify mischaracterizations, recalibrate their own self-ratings, and negotiate context-appropriate representations. We discuss implications for building inspectable, individually scoped communication profiles that let individuals control how agents represent them at work.

Mimetic Alignment with ASPECT: Evaluation of AI-inferred Personal Profiles

Abstract

AI agents that communicate on behalf of individuals need to capture how each person actually communicates, yet current approaches either require costly per-person fine-tuning, produce generic outputs from shallow persona descriptions, or optimize preferences without modeling communication style. We present ASPECT (Automated Social Psychometric Evaluation of Communication Traits), a pipeline that directs LLMs to assess constructs from a validated communication scale against behavioral evidence from workplace data, without per-person training. In a case study with 20 participants (1,840 paired item ratings, 600 scenario evaluations), ASPECT-generated profiles achieved moderate alignment with self-assessments, and ASPECT-generated responses were preferred over generic and self-report baselines on aggregate, with substantial variation across individuals and scenarios. During the profile review phase, linked evidence helped participants identify mischaracterizations, recalibrate their own self-ratings, and negotiate context-appropriate representations. We discuss implications for building inspectable, individually scoped communication profiles that let individuals control how agents represent them at work.

Paper Structure

This paper contains 60 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. From communication support to AI representation
  4. Building structured profiles of individuals
  5. System Design
  6. Conceptual Model
  7. Profile Pipeline Development
  8. Phase 1: Construct-guided Evidence Extraction.
  9. Phase 2: Inferred Assessment from Item-level Scoring.
  10. Phase 3: User Self-Assessment
  11. User Profile Evaluation
  12. Phase 4: User Profile Review
  13. Phase 5: User Scenario-Based Evaluation
  14. Study Design
  15. Evaluation in a Workplace Setting
  16. ...and 45 more sections

Figures (5)

  • Figure 1: Profile Generation Pipeline and User Evaluation Overview. See text for detailed explanation for each phase
  • Figure 2: Detailed outline of Phase 4: User Profile Review.
  • Figure 3: Scenario-based assessment interface
  • Figure 4: Win margin of each condition across participants and 10 scenarios.
  • Figure 5: Distribution of occurrences of facets with no examples found in data across 20 participants. Each facet is assessed once, so the maximum number of occurrences would be 20 per facet. This is evident of what traits tend to be lack of behavioral evidence from workplace data.