Header

UZH-Logo

Maintenance Infos

From monolithic to component-based performance evaluation of software architectures. A series of experiments analysing accuracy and effort


Martens, Anne; Koziolek, Heiko; Prechelt, Lutz; Reussner, Ralf (2011). From monolithic to component-based performance evaluation of software architectures. A series of experiments analysing accuracy and effort. Empirical Software Engineering, 16(5):587-622.

Abstract

Background: Model-based performance evaluation methods for softwarearchitectures can help architects to assess design alternatives andsave costs for late life-cycle performance fixes. A recent trendis component-based performance modelling, which aims at creatingreusable performance models; a number of such methods have been proposedduring the last decade. Their accuracy and the needed effort formodelling are heavily influenced by human factors, which are so farhardly understood empirically. Objective: Do component-based methods allow to make performance predictions with a comparable accuracy while saving effort in a reuse scenario? We examined three monolithic methods (SPE, umlPSI, Capacity Planning (CP)) and one component-based performance evaluation method (PCM) with regard to their accuracy and effort from the viewpoint of method users.Methods: We conducted a series of three experiments (with different levels of control) involving 47 computer science students. In the first experiment, we compared the applicability of the monolithic methods in order to choose one of them for comparison. In the second experiment, we compared the accuracy and effort of this monolithic and the component-based method for the model creation case. In the third, we studied the effort reduction from reusing component-based models. Data were collected based on the resulting artefacts, questionnaires and screen recording. They were analysed using hypothesis testing, linear models, and analysis of variance.Results: For the monolithic methods, we found that using SPE and CP resulted in accurate predictions, while umlPSI produced over-estimates. Comparing the component-based method PCM with SPE, we found that creating reusable models using PCM takes more (but not drastically more) time than using SPE and that participants can create accurate models with both techniques. Finally, we found that reusing PCM models can save time, because effort to reuse can be explained by a model that is independent of the inner complexity of a component.Limitations: The tasks performed in our experiments reflect only a subset of the actual activities when applying model-based performance evaluation methods in a software development process.Conclusions: Our results indicate that sufficient prediction accuracy can be achieved with both monolithic and component-based methods, and that the higher effort for component-based performance modelling will indeed pay off when the component models incorporate and hide a sufficient amount of complexity.

Abstract

Background: Model-based performance evaluation methods for softwarearchitectures can help architects to assess design alternatives andsave costs for late life-cycle performance fixes. A recent trendis component-based performance modelling, which aims at creatingreusable performance models; a number of such methods have been proposedduring the last decade. Their accuracy and the needed effort formodelling are heavily influenced by human factors, which are so farhardly understood empirically. Objective: Do component-based methods allow to make performance predictions with a comparable accuracy while saving effort in a reuse scenario? We examined three monolithic methods (SPE, umlPSI, Capacity Planning (CP)) and one component-based performance evaluation method (PCM) with regard to their accuracy and effort from the viewpoint of method users.Methods: We conducted a series of three experiments (with different levels of control) involving 47 computer science students. In the first experiment, we compared the applicability of the monolithic methods in order to choose one of them for comparison. In the second experiment, we compared the accuracy and effort of this monolithic and the component-based method for the model creation case. In the third, we studied the effort reduction from reusing component-based models. Data were collected based on the resulting artefacts, questionnaires and screen recording. They were analysed using hypothesis testing, linear models, and analysis of variance.Results: For the monolithic methods, we found that using SPE and CP resulted in accurate predictions, while umlPSI produced over-estimates. Comparing the component-based method PCM with SPE, we found that creating reusable models using PCM takes more (but not drastically more) time than using SPE and that participants can create accurate models with both techniques. Finally, we found that reusing PCM models can save time, because effort to reuse can be explained by a model that is independent of the inner complexity of a component.Limitations: The tasks performed in our experiments reflect only a subset of the actual activities when applying model-based performance evaluation methods in a software development process.Conclusions: Our results indicate that sufficient prediction accuracy can be achieved with both monolithic and component-based methods, and that the higher effort for component-based performance modelling will indeed pay off when the component models incorporate and hide a sufficient amount of complexity.

Statistics

Citations

Dimensions.ai Metrics
5 citations in Web of Science®
15 citations in Scopus®
Google Scholar™

Altmetrics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:03 Faculty of Economics > Department of Informatics
Dewey Decimal Classification:000 Computer science, knowledge & systems
Language:English
Date:2011
Deposited On:09 Feb 2012 16:28
Last Modified:17 Feb 2018 14:43
Publisher:Springer
ISSN:1382-3256 (P) 1573-7616 (E)
OA Status:Closed
Publisher DOI:https://doi.org/10.1007/s10664-010-9142-8
Other Identification Number:merlin-id:3889

Download

Full text not available from this repository.
View at publisher