Header

UZH-Logo

Maintenance Infos

Understanding flaky tests: the developer’s perspective


Eck, Moritz; Palomba, Fabio; Castelluccio, Marco; Bacchelli, Alberto (2019). Understanding flaky tests: the developer’s perspective. In: ESEC/FSE '19: 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Tallinn Estonia, 26 September 2019 - 30 September 2019, 830-840.

Abstract

Flaky tests are software tests that exhibit a seemingly random outcome (pass or fail) despite exercising unchanged code. In this work, we examine the perceptions of software developers about the nature, relevance, and challenges of flaky tests.

We asked 21 professional developers to classify 200 flaky tests they previously fixed, in terms of the nature and the origin of the flakiness, as well as of the fixing effort. We also examined developers' fixing strategies. Subsequently, we conducted an online survey with 121 developers with a median industrial programming experience of five years. Our research shows that: The flakiness is due to several different causes, four of which have never been reported before, despite being the most costly to fix; flakiness is perceived as significant by the vast majority of developers, regardless of their team's size and project's domain, and it can have effects on resource allocation, scheduling, and the perceived reliability of the test suite; and the challenges developers report to face regard mostly the reproduction of the flaky behavior and the identification of the cause for the flakiness. Public preprint [http://arxiv.org/abs/1907.01466], data and materials [https://doi.org/10.5281/zenodo.3265785].

Abstract

Flaky tests are software tests that exhibit a seemingly random outcome (pass or fail) despite exercising unchanged code. In this work, we examine the perceptions of software developers about the nature, relevance, and challenges of flaky tests.

We asked 21 professional developers to classify 200 flaky tests they previously fixed, in terms of the nature and the origin of the flakiness, as well as of the fixing effort. We also examined developers' fixing strategies. Subsequently, we conducted an online survey with 121 developers with a median industrial programming experience of five years. Our research shows that: The flakiness is due to several different causes, four of which have never been reported before, despite being the most costly to fix; flakiness is perceived as significant by the vast majority of developers, regardless of their team's size and project's domain, and it can have effects on resource allocation, scheduling, and the perceived reliability of the test suite; and the challenges developers report to face regard mostly the reproduction of the flaky behavior and the identification of the cause for the flakiness. Public preprint [http://arxiv.org/abs/1907.01466], data and materials [https://doi.org/10.5281/zenodo.3265785].

Statistics

Citations

Dimensions.ai Metrics
1 citation in Web of Science®
11 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

4 downloads since deposited on 26 Jan 2021
4 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Paper), refereed, original work
Communities & Collections:03 Faculty of Economics > Department of Informatics
Dewey Decimal Classification:000 Computer science, knowledge & systems
Scopus Subject Areas:Physical Sciences > Artificial Intelligence
Physical Sciences > Software
Language:English
Event End Date:30 September 2019
Deposited On:26 Jan 2021 10:27
Last Modified:27 Jan 2021 21:02
Publisher:ACM
ISBN:9781450355728
OA Status:Green
Publisher DOI:https://doi.org/10.1145/3338906.3338945
Other Identification Number:merlin-id:20225

Download

Green Open Access

Download PDF  'Understanding flaky tests: the developer’s perspective'.
Preview
Content: Published Version
Filetype: PDF
Size: 761kB
View at publisher