Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Artifact Validity in Design Science Research (DSR): A Comparative Analysis of Three Influential Frameworks

Sylvana Kroop

TL;DR

Design Science Research (DSR) faces questions about artifact validity as artifacts are applied across contexts. The paper compares three influential DSR frameworks (Hevner et al., Peffers et al., and Österle/Benner-Wickner integrated framework) through a five-type validity lens to assess support for validity. It proposes a revised DSR framework that formalizes Instrument, Technical, Design, Purpose, and Generalization validity with definitions, examples, and scoring guidelines, and evaluates existing frameworks against this rubric. The study finds Purpose validity is consistently strong, while Instrument and Design validity remain underdeveloped and risk invalid artifacts; the revised framework aims to improve rigor and systematic artifact evaluation. The work has practical impact for DSR education and practice by enabling more robust, transferable, and credible artifact evaluations and guiding future methodological enhancements.

Abstract

Although the methodology of Design Science Research (DSR) is playing an increasingly important role with the emergence of the "sciences of the artificial", the validity of the resulting artifacts is occasionally questioned. This paper compares three influential DSR frameworks to assess their support for artifact validity. Using five essential validity types (instrument validity, technical validity, design validity, purpose validity and generalization), the qualitative analysis reveals that while purpose validity is explicitly emphasized, instrument and design validity remain the least developed. Their implicit treatment in all frameworks poses a risk of overlooked validation, and the absence of mandatory instrument validity can lead to invalid artifacts, threatening research credibility. Beyond these findings, the paper contributes (a) a comparative overview of each framework's strengths and weaknesses and (b) a revised DSR framework incorporating all five validity types with definitions and examples. This ensures systematic artifact evaluation and improvement, reinforcing the rigor of DSR.

Artifact Validity in Design Science Research (DSR): A Comparative Analysis of Three Influential Frameworks

TL;DR

Design Science Research (DSR) faces questions about artifact validity as artifacts are applied across contexts. The paper compares three influential DSR frameworks (Hevner et al., Peffers et al., and Österle/Benner-Wickner integrated framework) through a five-type validity lens to assess support for validity. It proposes a revised DSR framework that formalizes Instrument, Technical, Design, Purpose, and Generalization validity with definitions, examples, and scoring guidelines, and evaluates existing frameworks against this rubric. The study finds Purpose validity is consistently strong, while Instrument and Design validity remain underdeveloped and risk invalid artifacts; the revised framework aims to improve rigor and systematic artifact evaluation. The work has practical impact for DSR education and practice by enabling more robust, transferable, and credible artifact evaluations and guiding future methodological enhancements.

Abstract

Although the methodology of Design Science Research (DSR) is playing an increasingly important role with the emergence of the "sciences of the artificial", the validity of the resulting artifacts is occasionally questioned. This paper compares three influential DSR frameworks to assess their support for artifact validity. Using five essential validity types (instrument validity, technical validity, design validity, purpose validity and generalization), the qualitative analysis reveals that while purpose validity is explicitly emphasized, instrument and design validity remain the least developed. Their implicit treatment in all frameworks poses a risk of overlooked validation, and the absence of mandatory instrument validity can lead to invalid artifacts, threatening research credibility. Beyond these findings, the paper contributes (a) a comparative overview of each framework's strengths and weaknesses and (b) a revised DSR framework incorporating all five validity types with definitions and examples. This ensures systematic artifact evaluation and improvement, reinforcing the rigor of DSR.

Paper Structure

This paper contains 16 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Methodology
  3. Related Work
  4. Revised DSR Framework focused on Artifact Validity
  5. Revised DSR Framework
  6. Validity Types
  7. Mandatory and Flexible Validity Types
  8. Scoring Guidelines for the Five Validity Types in DSR
  9. Comparative Evaluation of DSR Frameworks
  10. Instruments Validity
  11. Technical validity
  12. Design validity
  13. Purpose validity
  14. Generalization (external validity)
  15. Summarized assessment of the five validity types in the three DSR frameworks
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Type of validity categories used a priori for qualitative content analysis
  • Figure 2: Type of validity categories, aligned with frequently cited DSR literature on artifact evaluation
  • Figure 3: Revised DSR framework focused on artifact validity, based on the integrated DSR framework combining Österle et al. osterle_memorandum_2010 and Benner-Wickner et al. benner-wickner_leitfaden_2020
  • Figure 4: Rubric (scoring grid)
  • Figure 5: Consideration of the five validity types in three DSR frameworks