Better Assumptions, Stronger Conclusions: The Case for Ordinal Regression in HCI

TL;DR

This paper argues that ordinal data analyses in HCI currently suffer from inconsistent practices and over-reliance on metric-based methods. It advocates for ordinal regression via cumulative link (mixed) models (CL(M)Ms) to respect inherent ordering without assuming equal intervals, and illustrates their application with open CHI datasets using R. Through a systematic CHI 2024 literature review and practical CLM/CLMM re-analyses, the authors demonstrate that CL(M)Ms can reveal effects missed by traditional ANOVA and post-hoc tests, supporting more robust, reproducible inferences. The work contributes methodological guidance, accessible tooling, and concrete examples to help HCI researchers adopt ordinal-first modelling for more reliable insights and cross-study comparability.

Abstract

Despite the widespread use of ordinal measures in HCI, such as Likert-items, there is little consensus among HCI researchers on the statistical methods used for analysing such data. Both parametric and non-parametric methods have been extensively used within the discipline, with limited reflection on their assumptions and appropriateness for such analyses. In this paper, we examine recent HCI works that report statistical analyses of ordinal measures. We highlight prevalent methods used, discuss their limitations and spotlight key assumptions and oversights that diminish the insights drawn from these methods. Finally, we champion and detail the use of cumulative link (mixed) models (CLM/CLMM) for analysing ordinal data. Further, we provide practical worked examples of applying CLM/CLMMs using R to published open-sourced datasets. This work contributes towards a better understanding of the statistical methods used to analyse ordinal data in HCI and helps to consolidate practices for future work.

Figures (11)

• Figure 1: Overview of our review process detailing each step reported as per the PRISMA guidelines moher2009preferred. We randomly sampled 100 papers from our initial search result of 558 papers. The full text of all 100 papers was screened for eligibility. Six (6) papers were excluded, and the remaining 94 papers were included in our final sample.
• Figure 2: Frequency of different predictive models reportedly used for ordinal data analysis in our sample.
• Figure 3: Frequency of different omnibus tests used for ordinal data analysis in our sample.
• Figure 4: Frequency of different pairwise comparisons reportedly used for ordinal data analysis in our sample.
• Figure 5: Sankey diagram showing the progression of statistical procedures applied to constructs measured with ordinal data. Methods are displayed in sequence from predictive models (left), to omnibus tests (centre), to pairwise tests (right). Bars labelled ‘No’ indicate constructs for which no method was reported in that category. Red nodes indicate statistical methods that impose incompatible assumptions with ordinal data, blue nodes represent ordinal data compatible methods, yellow nodes represent methods whose appropriateness for analysing ordinal data is dependent on specific factors (such as the link' function used), and grey nodes represent absent methods in that specific category. As seen in the Figure, our sample reported limited use of predictive models (No model reported), and most models and omnibus approaches were not frequently used together (most models have links to 'No omnibus test reported'). The Figure also clearly highlights a lack of consensus within the field; with no clear direction of which method, or sequence of methods, should be used to analyse ordinal data.