Header

UZH-Logo

Maintenance Infos

Evaluating Learning-to-Rank Models for Prioritizing Code Review Requests using Process Simulation


Yang, Lanxin; Liu, Bohan; Jia, Junyu; Xue, Junming; Xu, Jinwei; Bacchelli, Alberto; Zhang, He (2023). Evaluating Learning-to-Rank Models for Prioritizing Code Review Requests using Process Simulation. In: 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), Taipa, Macao, 21 March 2023 - 24 March 2023. Institute of Electrical and Electronics Engineers, 461-472.

Abstract

In large-scale, active software projects, one of the main challenges with code review is prioritizing the many Code Review Requests (CRRs) these projects receive. Prior studies have developed many Learning-to-Rank (LtR) models in support of prioritizing CRRs and adopted rich evaluation metrics to compare their performances. However, the evaluation was performed before observing the complex interactions between CRRs and reviewers, activities and activities in real-world code reviews. Such a pre-review evaluation provides few indications about how effective LtR models contribute to code reviews. This study aims to perform a post-review evaluation on LtR models for prioritizing CRRs. To establish the evaluation environment, we employ Discrete-Event Simulation (DES) paradigm-based Software Process Simulation Modeling (SPSM) to simulate real-world code review processes, together with three customized evaluation metrics. We develop seven LtR models and use the historical review orders of CRRs as baselines for evaluation. The results indicate that employing LtR can effectively help to accelerate the completion of reviewing CRRs and the delivery of qualified code changes. Among the seven LtR models, LambdaMART and AdaRank are particularly beneficial for accelerating completion and delivery, respectively. This study empirically demonstrates the effectiveness of using DES-based SPSM for simulating code review processes, the benefits of using LtR for prioritizing CRRs, and the specific advantages of several LtR models. This study provides new ideas for software organizations that seek to evaluate LtR models and other artificial intelligence-powered software techniques.

Abstract

In large-scale, active software projects, one of the main challenges with code review is prioritizing the many Code Review Requests (CRRs) these projects receive. Prior studies have developed many Learning-to-Rank (LtR) models in support of prioritizing CRRs and adopted rich evaluation metrics to compare their performances. However, the evaluation was performed before observing the complex interactions between CRRs and reviewers, activities and activities in real-world code reviews. Such a pre-review evaluation provides few indications about how effective LtR models contribute to code reviews. This study aims to perform a post-review evaluation on LtR models for prioritizing CRRs. To establish the evaluation environment, we employ Discrete-Event Simulation (DES) paradigm-based Software Process Simulation Modeling (SPSM) to simulate real-world code review processes, together with three customized evaluation metrics. We develop seven LtR models and use the historical review orders of CRRs as baselines for evaluation. The results indicate that employing LtR can effectively help to accelerate the completion of reviewing CRRs and the delivery of qualified code changes. Among the seven LtR models, LambdaMART and AdaRank are particularly beneficial for accelerating completion and delivery, respectively. This study empirically demonstrates the effectiveness of using DES-based SPSM for simulating code review processes, the benefits of using LtR for prioritizing CRRs, and the specific advantages of several LtR models. This study provides new ideas for software organizations that seek to evaluate LtR models and other artificial intelligence-powered software techniques.

Statistics

Citations

Dimensions.ai Metrics

Altmetrics

Downloads

10 downloads since deposited on 19 Feb 2024
10 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Paper), not_refereed, original work
Communities & Collections:03 Faculty of Economics > Department of Informatics
Dewey Decimal Classification:000 Computer science, knowledge & systems
Scopus Subject Areas:Physical Sciences > Software
Physical Sciences > Safety, Risk, Reliability and Quality
Scope:Discipline-based scholarship (basic research)
Language:English
Event End Date:24 March 2023
Deposited On:19 Feb 2024 13:35
Last Modified:20 Jun 2024 13:38
Publisher:Institute of Electrical and Electronics Engineers
Series Name:IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)
ISSN:2640-7574
ISBN:978-1-6654-5278-6
OA Status:Green
Publisher DOI:https://doi.org/10.1109/saner56733.2023.00050
Related URLs:https://figshare.com/s/a033e99cd2a61e64c8bc (Research Data)
Project Information:
  • : FunderSNSF
  • : Grant ID200021_197227
  • : Project TitleEnhanced Code Review: Using Context and Learning from Review Experience
  • : FunderResearch and Development
  • : Grant ID
  • : Project Title
  • : FunderResearch and Development
  • : Grant ID
  • : Project Title
  • : FunderNational Science Foundation
  • : Grant ID
  • : Project Title
  • Content: Accepted Version
  • Language: English
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)