GuideGen: An Approach for Keeping Requirements and Acceptance Tests Aligned via Automatically Generated Guidance

Abstract

Context: When software-based systems evolve, their requirements change. The changes in requirements affect the associated acceptance tests, which should be adapted accordingly. In practice, however, requirements and their acceptance tests are not always kept up-to-date nor aligned. Such inconsistencies may introduce software quality problems, unintended costs and project delays.
Objective: In order to keep evolving requirements and their acceptance tests aligned, we are developing an approach called GuideGen. GuideGen automatically generates guidance in natural language about how to adapt the impacted acceptance tests when their requirements change.
Method: We have implemented GuideGen as a prototype tool and evaluated it in two studies: first, by assessing the correctness, completeness, understandability and relevance of the generated guidance using three data sets from industry and second, by assessing the applicability and usefulness of the approach and the tool with 23 practitioners from ten companies. When a requirement having more than one associated acceptance test is changed, GuideGen currently generates guidance for all of them together. As a first step towards overcoming this limitation, we assessed how well existing methods for change impact analysis can identify the tests actually impacted by the changes in a requirement.
Results: In the first study, we found that GuideGen produced correct guidance in about 67 to 89 percent of all changes. Our approach performed better for agile requirements than for traditional ones. The results of the second study show that GuideGen is perceived to be useful, but that the practitioners would prefer a GuideGen plug-in for commercial tools instead of a standalone tool. Further, in our experiment we could correctly identify the affected acceptance tests for 63% to 91% of the changes in the requirements.
Conclusion: Our approach facilitates the alignment of acceptance tests with the actual requirements and can improve the communication between requirements engineers and testers.