Reflection Before Action: Designing a Framework for Quantifying Thought Patterns for Increased Self-awareness in Personal Decision Making

TL;DR

The paper defines Pre-Decision Reflection (PDR) and introduces PROBE, a framework that quantifies pre-decision thinking along breadth (diversity of thought categories) and depth (elaboration). Through formative and summative studies with 46 participants, PROBE demonstrates reliable measurement of reflection patterns and reveals substantial heterogeneity across individuals, including a notable mismatch between self-perceived and observed depth/breadth. Findings indicate that most participants’ reflections are shallow and that many would benefit from supportive tools, while a subset show deep but narrow engagement. The work discusses implications for decision-support systems and agentic AI to surface hidden thought patterns and support meta-cognitive awareness, while acknowledging limitations and outlining directions for expanding validation and automatic category detection.

Abstract

When making significant life decisions, people increasingly turn to conversational AI tools, such as large language models (LLMs). However, LLMs often steer users toward solutions, limiting metacognitive awareness of their own decision-making. In this paper, we shift the focus in decision support from solution-orientation to reflective activity, coining the term pre-decision reflection (PDR). We introduce PROBE, the first framework that assesses pre-decision reflections along two dimensions: breadth (diversity of thought categories) and depth (elaborateness of reasoning). Coder agreement demonstrates PROBE's reliability in capturing how people engage in pre-decision reflection. Our study reveals substantial heterogeneity across participants and shows that people perceived their unassisted reflections as deeper and broader than PROBE's measures. By surfacing hidden thought patterns, PROBE opens opportunities for technologies that foster self-awareness and strengthen people's agency in choosing which thought patterns to rely on in decision-making.

Figures (7)

• Figure 1: Overview of the research study. The flowchart illustrates the progression from the Formative Study to the Summative Study. In the Formative Study, six participants’ pre-decision reflections (gathered through simple conversations with an agent) and the initial framework (PROBE V0) underwent iterative coding and refinement, during which PROBE categories were retained, slightly revised, or newly introduced. This process resulted in the refined framework (PROBE V1). The Summative Study then applied this framework to a larger set of participant reflections ($n=40$), which revealed diverse reflection patterns across participants. Together, the two phases highlight the process of framework development and evaluation, where empirical data informed refinement, and refined framework enabled systematic measurement.
• Figure 2: Distribution of SRIS scores collected from participants ($n=40$) of the summative study. It is roughly bell-shaped, peaking between scores of 45 and 50. The mean score was 46.12 (SD = 8.28) on a scale with a maximum of 72.
• Figure 3: Frequency of decision categories. The bar chart depicts the frequency of each decision category in the summative study data. "Career" was chosen by 7 participants, "Finances" by 11, "Family" by 13, and "Relocation" by 9. The overall distribution suggests that the data is diverse in terms of decision topics.
• Figure 4: Pre-decision Reflective Thought Patterns across participants from the summative study. The heatmap shows the distribution each participant's reflection across the seven PROBE categories, and the bar chart shows the depth of reflections for the same participants, defined as the percentage of elaborated thoughts among all their reflective thoughts. Columns represent participants, sorted by the variability of their reflections across categories (low variation on the left, high variation on the right), and rows are ordered by overall prevalence across all participants (most frequent at the bottom). The plots highlight the diversity of reflective patterns across participants, including distinctive cases such as P25, who exhibits very low diversity but high depth; P2, who shows high diversification but low elaboration; and P16, who shows not only very low diversification but also no elaboration.
• Figure 5: Distribution of participants by their self-assessments of the breadth and depth of their own reflections on a five-point Likert scale. The size of the circles indicate the number of participants in the corresponding depth vs. breadth "bin". Quotes from participants' self-assessments of their reflection show a focus on the process of reflection, but lack specifics on the quality and aspects of reflection.