Analysing academic paper ranking algorithms using test data and benchmarks: an investigation

Yu Zhang, Min Wang, Morteza Saberi, Elizabeth Chang

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)
3 Downloads (Pure)


Research on academic paper ranking has received great attention in recent years, and many algorithms have been proposed to automatically assess a large number of papers for this purpose. How to evaluate or analyse the performance of these ranking algorithms becomes an open research question. Theoretically, evaluation of an algorithm requires to compare its ranking result against a ground truth paper list. However, such ground truth does not exist in the field of scholarly ranking due to the fact that there does not and will not exist an absolutely unbiased, objective, and unified standard to formulate the impact of papers. Therefore, in practice researchers evaluate or analyse their proposed ranking algorithms by different methods, such as using domain expert decisions (test data) and comparing against predefined ranking benchmarks. The question is whether using different methods leads to different analysis results, and if so, how should we analyse the performance of the ranking algorithms? To answer these questions, this study compares among test data and different citation-based benchmarks by examining their relationships and assessing the effect of the method choices on their analysis results. The results of our experiments show that there does exist difference in analysis results when employing test data and different benchmarks, and relying exclusively on one benchmark or test data may bring inadequate analysis results. In addition, a guideline on how to conduct a comprehensive analysis using multiple benchmarks from different perspectives is summarised, which can help provide a systematic understanding and profile of the analysed algorithms.

Original languageEnglish
Pages (from-to)4045-4074
Number of pages30
Issue number7
Publication statusPublished - Jun 2022
Externally publishedYes


Dive into the research topics of 'Analysing academic paper ranking algorithms using test data and benchmarks: an investigation'. Together they form a unique fingerprint.

Cite this