Header

UZH-Logo

Maintenance Infos

An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment


Laaber, Christoph; Leitner, Philipp (2018). An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment. In: MSR ’18: 15th International Conference on Mining Software Repositories, Gothenburg, Sweden, 28 May 2018 - 29 May 2018.

Abstract

Continuous integration (CI) emphasizes quick feedback to devel- opers. This is at odds with current practice of performance testing, which predominantely focuses on long-running tests against entire systems in production-like environments. Alternatively, software microbenchmarking attempts to establish a performance baseline for small code fragments in short time. This paper investigates the quality of microbenchmark suites with a focus on suitability to deliver quick performance feedback and CI integration. We study ten open-source libraries written in Java and Go with benchmark suite sizes ranging from 16 to 983 tests, and runtimes between 11 minutes and 8.75 hours. We show that our study subjects include benchmarks with result variability of 50% or higher, indicating that not all benchmarks are useful for reliable discovery of slow- downs. We further arti cially inject actual slowdowns into public API methods of the study subjects and test whether test suites are able to discover them. We introduce a performance-test quality metric called the API benchmarking score (ABS). ABS represents a benchmark suite’s ability to nd slowdowns among a set of de ned core API methods. Resulting benchmarking scores (i.e., fraction of discovered slowdowns) vary between 10% and 100% for the study subjects. This paper’s methodology and results can be used to (1) assess the quality of existing microbenchmark suites, (2) select a set of tests to be run as part of CI, and (3) suggest or generate benchmarks for currently untested parts of an API.

Abstract

Continuous integration (CI) emphasizes quick feedback to devel- opers. This is at odds with current practice of performance testing, which predominantely focuses on long-running tests against entire systems in production-like environments. Alternatively, software microbenchmarking attempts to establish a performance baseline for small code fragments in short time. This paper investigates the quality of microbenchmark suites with a focus on suitability to deliver quick performance feedback and CI integration. We study ten open-source libraries written in Java and Go with benchmark suite sizes ranging from 16 to 983 tests, and runtimes between 11 minutes and 8.75 hours. We show that our study subjects include benchmarks with result variability of 50% or higher, indicating that not all benchmarks are useful for reliable discovery of slow- downs. We further arti cially inject actual slowdowns into public API methods of the study subjects and test whether test suites are able to discover them. We introduce a performance-test quality metric called the API benchmarking score (ABS). ABS represents a benchmark suite’s ability to nd slowdowns among a set of de ned core API methods. Resulting benchmarking scores (i.e., fraction of discovered slowdowns) vary between 10% and 100% for the study subjects. This paper’s methodology and results can be used to (1) assess the quality of existing microbenchmark suites, (2) select a set of tests to be run as part of CI, and (3) suggest or generate benchmarks for currently untested parts of an API.

Statistics

Citations

Dimensions.ai Metrics

Altmetrics

Downloads

29 downloads since deposited on 04 Dec 2018
29 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Paper), refereed, original work
Communities & Collections:03 Faculty of Economics > Department of Informatics
Dewey Decimal Classification:000 Computer science, knowledge & systems
Language:English
Event End Date:29 May 2018
Deposited On:04 Dec 2018 16:23
Last Modified:04 Dec 2018 19:30
Publisher:ACM
OA Status:Green
Free access at:Related URL. An embargo period may apply.
Publisher DOI:https://doi.org/10.1145/3196398.3196407
Related URLs:http://www.ifi.uzh.ch/dam/jcr:ccf1399a-2d57-4ff9-a3b0-59d69616d5d3/msr18-author-version.pdf (Author)
Other Identification Number:merlin-id:16259

Download

Download PDF  'An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment'.
Preview
Content: Published Version
Filetype: PDF
Size: 1MB
View at publisher