In the past decades, many countries have started to fund academic institutions based on the evaluation of their scientific performance. In this context, post-publication peer review is often used to assess scientific performance. Bibliometric indicators have been suggested as an alternative to peer review. A recurrent question in this context is whether peer review and metrics tend to yield similar outcomes. In this paper, we study the agreement between bibliometric indicators and peer review based on a sample of publications submitted for evaluation to the national Italian research assessment exercise (2011--2014). In particular, we study the agreement between bibliometric indicators and peer review at a higher aggregation level, namely the institutional level. Additionally, we also quantify the internal agreement of peer review at the institutional level. We base our analysis on a hierarchical Bayesian model using cross-validation. We find that the level of agreement is generally higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is on par with the internal agreement among two reviewers for certain fields of science in this particular context. This suggests that for some fields, bibliometric indicators may possibly be considered as an alternative to peer review for the Italian national research assessment exercise. Although results do not necessarily generalise to other contexts, it does raise the question whether similar findings would obtain for other research assessment exercises, such as in the United Kingdom.
翻译:在过去几十年中,许多国家开始根据学术机构的科研表现来资助它们。在这种情况下,通常使用后出版同行评审来评估科研表现。学術衡量指标则被视为同行评审的替代方案。在这种情况下,一个常见的问题是,同行评议和学術衡量指标是否有相似的结果。本文研究了意大利国家级研究评估项目(2011-2014期间)提交评估的一份出版物样本,以此来研究学術衡量指标和同行评议在更高层面上,即机构水平上的一致性。此外,我们还量化了机构水平下的同行评议内部一致性。我们利用交叉验证的层级贝叶斯模型进行了分析。我们发现,在机构水平上,指标和同行评议的一致性普遍高于出版物水平。总体而言,对于某些科学领域,指标与同行评议的一致性与两位审稿人的内部一致性大致相当。这表明,在某些领域,意大利国家级研究评估项目中,学術衡量指标可能被认为是对同行评议的替代方案。尽管结果不一定适用于其他情况,但它确实引发了一个问题,即类似的发现是否适用于其他研究评估项目,例如英国。