Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability

Inter-rater reliability (IRR) is one of the commonly used tools for assessing the quality of ratings from multiple raters as it is easily obtainable from the observed ratings themselves. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome; the applicant is either selected or not. This final outcome is not considered in IRR, which instead focuses on the ratings of the individual subjects or objects. In this work, we outline the connection between the ratings' measurement model (used for IRR) and a binary classification framework. We develop a quantile approximation which allows us to estimate the probability of correctly selecting the best applicants and compute error probabilities of the selection procedure (i.e., false-positive and false-negative rate) under the assumption of the ratings' validity. If the ratings are not completely valid, the computed error probabilities correspond to a lower bound on the true error probabilities. We draw connections between the inter-rater reliability and the binary classification metrics, showing that binary classification metrics depend solely on the IRR coefficient and proportion of selected applicants. We assess the performance of the quantile approximation in a simulation study and apply it in an example comparing the reliability of multiple grant peer review selection procedures.