Header

UZH-Logo

Maintenance Infos

Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality


Laaber, Christoph; Würsten, Stefan; Gall, Harald C; Leitner, Philipp (2020). Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality. In: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, 8 December 2020 - 13 December 2020, 989-1001.

Abstract

Executing software microbenchmarks, a form of small-scale performance tests predominantly used for libraries and frameworks, is a costly endeavor. Full benchmark suites take up to multiple hours or days to execute, rendering frequent checks, e.g., as part of continuous integration (CI), infeasible. However, altering benchmark configurations to reduce execution time without considering the impact on result quality can lead to benchmark results that are not representative of the software’s true performance.
We propose the first technique to dynamically stop software microbenchmark executions when their results are sufficiently stable. Our approach implements three statistical stoppage criteria and is capable of reducing Java Microbenchmark Harness (JMH) suite execution times by 48.4% to 86.0%. At the same time it retains the same result quality for 78.8% to 87.6% of the benchmarks, compared to executing the suite for the default duration.
The proposed approach does not require developers to manually craft custom benchmark configurations; instead, it provides automated mechanisms for dynamic reconfiguration. Hence, making dynamic reconfiguration highly effective and efficient, potentially paving the way to inclusion of JMH microbenchmarks in CI.

Abstract

Executing software microbenchmarks, a form of small-scale performance tests predominantly used for libraries and frameworks, is a costly endeavor. Full benchmark suites take up to multiple hours or days to execute, rendering frequent checks, e.g., as part of continuous integration (CI), infeasible. However, altering benchmark configurations to reduce execution time without considering the impact on result quality can lead to benchmark results that are not representative of the software’s true performance.
We propose the first technique to dynamically stop software microbenchmark executions when their results are sufficiently stable. Our approach implements three statistical stoppage criteria and is capable of reducing Java Microbenchmark Harness (JMH) suite execution times by 48.4% to 86.0%. At the same time it retains the same result quality for 78.8% to 87.6% of the benchmarks, compared to executing the suite for the default duration.
The proposed approach does not require developers to manually craft custom benchmark configurations; instead, it provides automated mechanisms for dynamic reconfiguration. Hence, making dynamic reconfiguration highly effective and efficient, potentially paving the way to inclusion of JMH microbenchmarks in CI.

Statistics

Citations

Dimensions.ai Metrics

Altmetrics

Downloads

16 downloads since deposited on 14 Dec 2020
16 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Paper), refereed, original work
Communities & Collections:03 Faculty of Economics > Department of Informatics
Dewey Decimal Classification:000 Computer science, knowledge & systems
Scopus Subject Areas:Physical Sciences > Software
Language:English
Event End Date:13 December 2020
Deposited On:14 Dec 2020 16:26
Last Modified:21 Jan 2021 18:45
Publisher:ACM
Series Name:ESEC/FSE '20
ISBN:9781450370431
OA Status:Green
Free access at:Related URL. An embargo period may apply.
Publisher DOI:https://doi.org/10.1145/3368089.3409683
Related URLs:http://t.uzh.ch/13k (Author)
Other Identification Number:merlin-id:20158

Download

Green Open Access

Download PDF  'Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality'.
Preview
Content: Published Version
Language: English
Filetype: PDF
Size: 1MB
View at publisher