Header

UZH-Logo

Maintenance Infos

Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data


Xu, Shuqi; Mariani, Manuel; Lü, Linyuan; Medo, Matúš (2020). Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data. Journal of Informetrics, 14(1):101005.

Abstract

Despite the increasing use of citation-based metrics for research evaluation purposes, we do not know yet which metrics best deliver on their promise to gauge the significance of a scientific paper or a patent. We assess 17 network-based metrics by their ability to identify milestone papers and patents in three large citation datasets. We find that traditional information-retrieval evaluation metrics are strongly affected by the interplay between the age distribution of the milestone items and age biases of the evaluated metrics. Outcomes of these metrics are therefore not representative of the metrics’ ranking ability. We argue in favor of a modified evaluation procedure that explicitly penalizes biased metrics and allows us to reveal metrics’ performance patterns that are consistent across the datasets. PageRank and LeaderRank turn out to be the best-performing ranking metrics when their age bias is suppressed by a simple transformation of the scores that they produce, whereas other popular metrics, including citation count, HITS and Collective Influence, produce significantly worse ranking results.

Abstract

Despite the increasing use of citation-based metrics for research evaluation purposes, we do not know yet which metrics best deliver on their promise to gauge the significance of a scientific paper or a patent. We assess 17 network-based metrics by their ability to identify milestone papers and patents in three large citation datasets. We find that traditional information-retrieval evaluation metrics are strongly affected by the interplay between the age distribution of the milestone items and age biases of the evaluated metrics. Outcomes of these metrics are therefore not representative of the metrics’ ranking ability. We argue in favor of a modified evaluation procedure that explicitly penalizes biased metrics and allows us to reveal metrics’ performance patterns that are consistent across the datasets. PageRank and LeaderRank turn out to be the best-performing ranking metrics when their age bias is suppressed by a simple transformation of the scores that they produce, whereas other popular metrics, including citation count, HITS and Collective Influence, produce significantly worse ranking results.

Statistics

Citations

Dimensions.ai Metrics
19 citations in Web of Science®
18 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

94 downloads since deposited on 20 Feb 2020
30 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:03 Faculty of Economics > Department of Business Administration
08 Research Priority Programs > Social Networks
Dewey Decimal Classification:330 Economics
Scopus Subject Areas:Physical Sciences > Computer Science Applications
Social Sciences & Humanities > Library and Information Sciences
Language:English
Date:1 February 2020
Deposited On:20 Feb 2020 10:47
Last Modified:23 Nov 2023 02:47
Publisher:Elsevier
ISSN:1751-1577
OA Status:Green
Publisher DOI:https://doi.org/10.1016/j.joi.2019.101005
Other Identification Number:merlin-id:19177
  • Content: Accepted Version
  • Language: English
  • Licence: Creative Commons: Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)