A Soundness and Precision Benchmark for Java Debloating Tools

TL;DR

This paper addresses the challenge of reducing Java dependency bloating while preserving runtime correctness. It introduces Deblometer, a micro-benchmark with 59 test cases across 13 Java language features and a validation harness that computes per-feature soundness and precision. Soundness $S=TP/(TP+FN)$ and precision $P=TP/(TP+FP)$ are reported per feature and debloating level. Evaluating Deptrim, JShrink, and ProGuard reveals distinct trade-offs: Deptrim is more sound but less precise, ProGuard is leaner but risks removing required constructs, and JShrink struggles with annotations and sometimes corrupts artifacts. Dynamic features like dynamic class loading and reflection pose the greatest unsoundness across tools, motivating more robust, possibly dynamic-analysis-informed debloating approaches. Deblometer provides a ground-truth-driven framework to systematically benchmark and guide improvements in Java debloating tools.

Abstract

Modern software development reuses code by importing libraries as dependencies. Software projects typically include an average of 36 dependencies, with 80% being transitive, meaning they are dependencies of dependencies. Recent research indicates that only 24.9% of these dependencies are required at runtime, and even within those, many program constructs remain unused, adding unnecessary code to the project. This has led to the development of debloating tools that remove unnecessary dependencies and program constructs while balancing precision by eliminating unused constructs and soundness by preserving all required constructs. To systematically evaluate this trade-off, we developed Deblometer, a micro-benchmark consisting of 59 test cases designed to assess support for various Java language features in debloating tools. Each test case includes a manually curated ground truth specifying necessary and bloated classes, methods, and fields, enabling precise measurement of soundness and precision. Using Deblometer, we evaluated three popular Java debloating tools: Deptrim, JShrink, and ProGuard. Our evaluation reveals that all tools remove required program constructs, which results in changed semantics or execution crashes. In particular, the dynamic class loading feature introduces unsoundness in all evaluated tools. Our comparison shows that Deptrim retains more bloated constructs, while ProGuard removes more required constructs. JShrink's soundness is significantly affected by limited support for annotations, which leads to corrupted debloated artifacts. These soundness issues highlight the need to improve debloating tools to ensure stable and reliable debloated software.

Figures (2)

• Figure 1: Overview of Deblometer
• Figure 2: The evaluation process using Deblometer