Organize, Then Vote: Exploring Cognitive Load in Quadratic Survey Interfaces

TL;DR

Quadratic Surveys promise more accurate preference elicitation, but their cognitive demands hinder practical adoption. The authors introduce a two-phase Organize-Then-Vote QS interface to scaffold preference construction and reduce cognitive load, evaluating it against a baseline single-phase text interface in a 2x2 lab study (short vs long surveys; 6 vs 24 options). Bayesian and qualitative analyses reveal that the two-phase design shifts cognitive effort toward deeper processing, reduces satisficing in long surveys, and modulates edit-distance and time-per-option in ways that suggest more deliberate decision-making, though statistical significance of overall load differences is mixed. The work provides design principles, practical recommendations, and open-source implementations to advance QS usability and adoption, with implications for budget-aware, preference-elicitation tools in multi-option decision contexts.

Abstract

Quadratic Surveys (QSs) elicit more accurate preferences than traditional methods like Likert-scale surveys. However, the cognitive load associated with QSs has hindered their adoption in digital surveys for collective decision-making. We introduce a two-phase "organize-then-vote" QS to reduce cognitive load. As interface design significantly impacts survey results and accuracy, our design scaffolds survey takers' decision-making while managing the cognitive load imposed by QS. In a 2x2 between-subject in-lab study on public resource allotment, we compared our interface with a traditional text interface across a QS with 6 (short) and 24 (long) options. Two-phase interface participants spent more time per option and exhibited shorter voting edit distances. We qualitatively observed shifts in cognitive effort from mechanical operations to constructing more comprehensive preferences. We conclude that this interface promoted deeper engagement, potentially reducing satisficing behaviors caused by cognitive overload in longer QSs. This research clarifies how human-centered design improves preference elicitation tools for collective decision-making.

Figures (37)

• Figure 1: The Two-Phase Interface: The interface consists of two phases. Survey respondents can navigate between phases using the top right button. In the organization phase, the interface presents one option at a time to the respondents, and they chose one of four positional choices: "Lean Positive", "Lean Neutral", "Lean Negative", or "Skip". Skipped options are hidden and can be evaluated later. The chosen options then appear below. Items can be dragged and dropped across categories or returned to the stack. In the voting phase, options are listed in the order of the four categories. When hovering over each option, respondents can select a vote for that option using a dropdown menu. Each dropdown menu contains the cost associated with the vote. A sort button allows ascending sorting within each category. A summary box tracks the remaining credit balance.
• Figure 2: A selection of two QV interfaces. The interface on the left was used in the first empirical QV research quarfoot2017quadratic. Little information is available about the software, except for an image from posner2018radical. The interface on the right is an open-sourced QV interface RadicalxChangeQuadraticvoting2024 forked from GitCoin gitcoinReadWhitepaperGitcoin, used by the RadicalxChange community radicalxchange. Both interfaces share the common elements with different visual representations.
• Figure 3: Alternative vote control. The click-based design (upper) mirrors traditional vote control used in other QV interfaces, where each click controls one vote. The wheel-based design (the latter two) allows control through both clicks and mouse wheel rotation.
• Figure 4: The text-based interface: This interface is based on the two-phase version but does not include the organization phase and lacks the drag-and-drop functionality. Options are randomly positioned.
• Figure 5: Demographic distributions: Age, Gender, and Ethnicity