Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aggregating empirical evidence from data strategy studies: a case on model quantization

Santiago del Rey, Paulo Sérgio Medeiros dos Santos, Guilherme Horta Travassos, Xavier Franch, Silverio Martínez-Fernández

TL;DR

The paper tackles the challenge of synthesizing evidence from data-strategy studies in software engineering by using model quantization in DL as a test case. It applies the Structured Synthesis Method (SSM) to six empirical studies, yielding 19 evidence models that integrate qualitative and quantitative findings. The results indicate substantial resource-efficiency gains from quantization (notably storage and GPU energy) with only modest accuracy trade-offs, though evidence across techniques remains fragmented. This work demonstrates the feasibility of SSM for data-strategy synthesis, offers methodological lessons for aggregating non-human-subject research, and points to future work on data-strategy-specific quality criteria and broader synthesis approaches.

Abstract

Background: As empirical software engineering evolves, more studies adopt data strategies$-$approaches that investigate digital artifacts such as models, source code, or system logs rather than relying on human subjects. Synthesizing results from such studies introduces new methodological challenges. Aims: This study assesses the effects of model quantization on correctness and resource efficiency in deep learning (DL) systems. Additionally, it explores the methodological implications of aggregating evidence from empirical studies that adopt data strategies. Method: We conducted a research synthesis of six primary studies that empirically evaluate model quantization. We applied the Structured Synthesis Method (SSM) to aggregate the findings, which combines qualitative and quantitative evidence through diagrammatic modeling. A total of 19 evidence models were extracted and aggregated. Results: The aggregated evidence indicates that model quantization weakly negatively affects correctness metrics while consistently improving resource efficiency metrics, including storage size, inference latency, and GPU energy consumption$-$a manageable trade-off for many DL deployment contexts. Evidence across quantization techniques remains fragmented, underscoring the need for more focused empirical studies per technique. Conclusions: Model quantization offers substantial efficiency benefits with minor trade-offs in correctness, making it a suitable optimization strategy for resource-constrained environments. This study also demonstrates the feasibility of using SSM to synthesize findings from data strategy-based research.

Aggregating empirical evidence from data strategy studies: a case on model quantization

TL;DR

The paper tackles the challenge of synthesizing evidence from data-strategy studies in software engineering by using model quantization in DL as a test case. It applies the Structured Synthesis Method (SSM) to six empirical studies, yielding 19 evidence models that integrate qualitative and quantitative findings. The results indicate substantial resource-efficiency gains from quantization (notably storage and GPU energy) with only modest accuracy trade-offs, though evidence across techniques remains fragmented. This work demonstrates the feasibility of SSM for data-strategy synthesis, offers methodological lessons for aggregating non-human-subject research, and points to future work on data-strategy-specific quality criteria and broader synthesis approaches.

Abstract

Background: As empirical software engineering evolves, more studies adopt data strategiesapproaches that investigate digital artifacts such as models, source code, or system logs rather than relying on human subjects. Synthesizing results from such studies introduces new methodological challenges. Aims: This study assesses the effects of model quantization on correctness and resource efficiency in deep learning (DL) systems. Additionally, it explores the methodological implications of aggregating evidence from empirical studies that adopt data strategies. Method: We conducted a research synthesis of six primary studies that empirically evaluate model quantization. We applied the Structured Synthesis Method (SSM) to aggregate the findings, which combines qualitative and quantitative evidence through diagrammatic modeling. A total of 19 evidence models were extracted and aggregated. Results: The aggregated evidence indicates that model quantization weakly negatively affects correctness metrics while consistently improving resource efficiency metrics, including storage size, inference latency, and GPU energy consumptiona manageable trade-off for many DL deployment contexts. Evidence across quantization techniques remains fragmented, underscoring the need for more focused empirical studies per technique. Conclusions: Model quantization offers substantial efficiency benefits with minor trade-offs in correctness, making it a suitable optimization strategy for resource-constrained environments. This study also demonstrates the feasibility of using SSM to synthesize findings from data strategy-based research.
Paper Structure (21 sections, 1 equation, 3 figures, 2 tables)

This paper contains 21 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Model quantization
  4. Aggregating evidence from data strategy studies
  5. Methodology
  6. SSM step 1: Selecting primary studies
  7. Search strategy
  8. Inclusion and exclusion criteria
  9. Study selection
  10. SSM step 2: Analysis and evidence representation
  11. SSM step 3: Evidence synthesis
  12. Representation of model quantization effects
  13. Results
  14. Effects of model quantization that increased their belief
  15. Effects of model quantization with unchanged belief
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Experiment principles of the study (adapted from Wohlin et al. wohlinExperimentationSoftwareEngineering2012).
  • Figure 2: Evidence model representing the results of https://evidencefactory.lens-ese.cos.ufrj.br/evidenceEditor/263147.
  • Figure 3: Adapted forest plot showing the effects of applying model quantization on dependent variables from all the extracted evidence from primary studies S1 to S6. Links to each evidence diagram are provided in the replication package.