Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What Impacts the Quality of the User Answers when Asked about the Current Context?

Ivano Bison, Haonan Zhao, Fausto Giunchiglia

TL;DR

The paper investigates what factors affect the quality of user-provided answers to questions about current context in mobile ESM deployments. It builds a causal model linking exogenous context (situational, temporal, computing) and endogenous factors (burden, mood, traits) to reaction time, completion time, and answer accuracy, using a two-week study with 58,340 observations from 158 students and applying MLM/Cox regression and MSEM/SEM. Key findings show reaction time as the dominant driver of answer quality, with context and mood/procrastination shaping both reaction and completion times and contributing to about 9.6% incorrect responses; the study provides practical recommendations to improve data quality in real-world ESM studies. The work highlights the importance of context-aware notification strategies and carefully designed observation windows to minimize errors in subjective self-reports gathered in-the-wild. Overall, it offers actionable guidelines for researchers conducting EMA/ESM studies to optimize response quality and reliability.

Abstract

Sensor data provide an objective view of reality but fail to capture the subjective motivations behind an individual's behavior. This latter information is crucial for learning about the various dimensions of the personal context, thus increasing predictability. The main limitation is the human input, which is often not of the quality that is needed. The work so far has focused on the usually high number of missing answers. The focus of this paper is on \textit{the number of mistakes} made when answering questions. Three are the main contributions of this paper. First, we show that the user's reaction time, i.e., the time before starting to respond, is the main cause of a low answer quality, where its effects are both direct and indirect, the latter relating to its impact on the completion time, i.e., the time taken to compile the response. Second, we identify the specific exogenous (e.g., the situational or temporal context) and endogenous (e.g., mood, personality traits) factors which have an influence on the reaction time, as well as on the completion time. Third, we show how reaction and completion time compose their effects on the answer quality. The paper concludes with a set of actionable recommendations.

What Impacts the Quality of the User Answers when Asked about the Current Context?

TL;DR

The paper investigates what factors affect the quality of user-provided answers to questions about current context in mobile ESM deployments. It builds a causal model linking exogenous context (situational, temporal, computing) and endogenous factors (burden, mood, traits) to reaction time, completion time, and answer accuracy, using a two-week study with 58,340 observations from 158 students and applying MLM/Cox regression and MSEM/SEM. Key findings show reaction time as the dominant driver of answer quality, with context and mood/procrastination shaping both reaction and completion times and contributing to about 9.6% incorrect responses; the study provides practical recommendations to improve data quality in real-world ESM studies. The work highlights the importance of context-aware notification strategies and carefully designed observation windows to minimize errors in subjective self-reports gathered in-the-wild. Overall, it offers actionable guidelines for researchers conducting EMA/ESM studies to optimize response quality and reliability.

Abstract

Sensor data provide an objective view of reality but fail to capture the subjective motivations behind an individual's behavior. This latter information is crucial for learning about the various dimensions of the personal context, thus increasing predictability. The main limitation is the human input, which is often not of the quality that is needed. The work so far has focused on the usually high number of missing answers. The focus of this paper is on \textit{the number of mistakes} made when answering questions. Three are the main contributions of this paper. First, we show that the user's reaction time, i.e., the time before starting to respond, is the main cause of a low answer quality, where its effects are both direct and indirect, the latter relating to its impact on the completion time, i.e., the time taken to compile the response. Second, we identify the specific exogenous (e.g., the situational or temporal context) and endogenous (e.g., mood, personality traits) factors which have an influence on the reaction time, as well as on the completion time. Third, we show how reaction and completion time compose their effects on the answer quality. The paper concludes with a set of actionable recommendations.
Paper Structure (21 sections, 16 figures, 2 tables)

This paper contains 21 sections, 16 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. The overall model
  5. Q1: The reaction time
  6. Q2: The completion time
  7. Q3: The chain of causal effects on the quality of the data
  8. Sample Recruitment and questionnaires scheduling
  9. The Statistical Models
  10. Analysis
  11. Q1: Reaction time
  12. Q2: Completion time
  13. Q3: Answer correctness
  14. Take-away lessons
  15. The role of exogenous and endogenous factors on the answer quality
  16. ...and 6 more sections

Figures (16)

  • Figure 1: The causal chain of events impacting the answer quality.
  • Figure 2: Time diary.
  • Figure 3: Hardware and software sensor data collected together with their sampling rate. "On change" means that the value of the sensor is collected only when it changes its value.
  • Figure 4: Which impact on the answer quality?
  • Figure 5: Chain of errors: Multilevel structural equation model and a Structural Equation model.(Note: Between square brackets are reported the standardised parameters (i.e., "Coef B") of the SEM model.)
  • ...and 11 more figures