Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Advocacy: A Design Space for Replication-Related Studies

Yiheng Liang, Kim Marriott, Helen C. Purchase

TL;DR

This work presents a supporting multi-dimensional design space framework within which replication experimental design decisions can be identified, categorized, compared and analyzed.

Abstract

The importance of replication is often discussed and advocated -- not only in the domains of visualization and HCI, but in all scientific areas. When replicating a study, design decisions need to be made with regards which aspects of the original study will remain the same and which will be altered. We present a supporting multi-dimensional design space framework within which such decisions can be identified, categorized, compared and analyzed. The framework treats replication experimental design as a pairwise comparison problem, and represents the design by four practical dimensions defined by three comparison levels. The design space is therefore a framework that can be used for both retrospective characterization and prospective planning. We provide worked examples, and relate our framework to other attempts at describing the scope of replication studies.

Beyond Advocacy: A Design Space for Replication-Related Studies

TL;DR

This work presents a supporting multi-dimensional design space framework within which replication experimental design decisions can be identified, categorized, compared and analyzed.

Abstract

The importance of replication is often discussed and advocated -- not only in the domains of visualization and HCI, but in all scientific areas. When replicating a study, design decisions need to be made with regards which aspects of the original study will remain the same and which will be altered. We present a supporting multi-dimensional design space framework within which such decisions can be identified, categorized, compared and analyzed. The framework treats replication experimental design as a pairwise comparison problem, and represents the design by four practical dimensions defined by three comparison levels. The design space is therefore a framework that can be used for both retrospective characterization and prospective planning. We provide worked examples, and relate our framework to other attempts at describing the scope of replication studies.
Paper Structure (13 sections, 1 equation, 1 figure, 1 table)

This paper contains 13 sections, 1 equation, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Scope and Terminology
  3. Related Work
  4. Replication Discourse
  5. Empirical Study Design
  6. Design Space Framework
  7. Rationale
  8. Component Space
  9. Comparison Scale
  10. Framework and Collapse rules
  11. Worked Examples
  12. Discussion
  13. Conclusion

Figures (1)

  • Figure 1: The replication-related design space framework, showing how components of a replication study can compare to those of the reference study. HTML]FDCDAC$D_{identical}$ cases (inappropriate) are described in §\ref{['sec:collapse_rules']}.