Bias evaluation in machine-learning based services (MLS) based on traditional algorithmic fairness notions that rely on comparative principles is practically difficult, making it necessary to rely on human auditor feedback. However, in spite of taking rigorous training on various comparative fairness notions, human auditors are known to disagree on various aspects of fairness notions in practice, making it difficult to collect reliable feedback. This paper offers a paradigm shift to the domain of algorithmic fairness via proposing a new fairness notion based on the principle of non-comparative justice. In contrary to traditional fairness notions where the outcomes of two individuals/groups are compared, our proposed notion compares the MLS' outcome with a desired outcome for each input. This desired outcome naturally describes a human auditor's expectation, and can be easily used to evaluate MLS on crowd-auditing platforms. We show that any MLS can be deemed fair from the perspective of comparative fairness (be it in terms of individual fairness, statistical parity, equal opportunity or calibration) if it is non-comparatively fair with respect to a fair auditor. We also show that the converse holds true in the context of individual fairness. Given that such an evaluation relies on the trustworthiness of the auditor, we also present an approach to identify fair and reliable auditors by estimating their biases with respect to a given set of sensitive attributes, as well as quantify the uncertainty in the estimation of biases within a given MLS. Furthermore, all of the above results are also validated on COMPAS, German credit and Adult Census Income datasets.
翻译:以基于比较原则的传统算法公平概念为基础的机器学习服务(MLS)的比值评价实际上很难,因此有必要依赖人类审计员的反馈;然而,尽管在各种比较公平概念方面进行了严格的培训,但已知人类审计员在实践中对公平概念的各个方面有不同意见,因此难以收集可靠的反馈;本文件根据非比较公正原则提出一个新的公平概念,从而向算法公平领域转变;与对两个个人/群体的结果进行比较的传统公平概念相反,我们提出的概念将MLS的结果与每项投入的预期结果进行比较。这一预期结果自然地描述了一个人类审计员的期望,并且很容易用来评价人群审计平台上的MLS。我们表明,从比较公平的角度(无论是个人公平、统计均等、机会均等还是校正)来看,任何MLS都可被视为公平的公平,如果给予个人公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、公平、有、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有价值、有偿、有偿、有价值、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有偿、有、有、有、有偿、有