Decisions impacting human lives are increasingly being made or assisted by automated decision-making algorithms. Many of these algorithms process personal data for predicting recidivism, credit risk analysis, identifying individuals using face recognition, and more. While potentially improving efficiency and effectiveness, such algorithms are not inherently free from bias, opaqueness, lack of explainability, maleficence, and the like. Given that the outcomes of these algorithms have a significant impact on individuals and society and are open to analysis and contestation after deployment, such issues must be accounted for before deployment. Formal audits are a way of ensuring algorithms meet the appropriate accountability standards. This work, based on an extensive analysis of the literature and an expert focus group study, proposes a unifying framework for a system accountability benchmark for formal audits of artificial intelligence-based decision-aiding systems. This work also proposes system cards to serve as scorecards presenting the outcomes of such audits. It consists of 56 criteria organized within a four-by-four matrix composed of rows focused on (i) data, (ii) model, (iii) code, (iv) system, and columns focused on (a) development, (b) assessment, (c) mitigation, and (d) assurance. The proposed system accountability benchmark reflects the state-of-the-art developments for accountable systems, serves as a checklist for algorithm audits, and paves the way for sequential work in future research.
翻译:许多这些算法处理个人数据,用于预测累犯、信用风险分析、利用面部识别个人等等。这项工作在广泛分析文献和专家重点小组研究的基础上,为人工智能辅助决策系统正式审计的系统问责基准提出了一个统一框架。这项工作还提出系统卡作为记分卡,介绍这类审计的结果。它包括56项标准,由四对四的矩阵组成,侧重于(一) 数据,(二) 模式,(四) 代码,(四) 侧重于(a) 发展,(b) 评估,(c) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) 评估,(d) ) 评估,(d) 评估,(d) ) 评估,(d) 评估,(d) 评估,(d) ) 评估,(d) 评估,(d) ) 评估,(d) 评估,(d) ) 评估,(d) ) ) 评估,(d) 评估,(d) 评估,(d) ),(d) ) 评估,(d),(d) ),(d),(d),(d),(d),(d),(d) ),(d) ),(用于,) ) ),(用于) ) ) ) ),(d),(d) ) ),(用于,(d) ),(d),(d),(d),(d) ),(d),(d),(a),(用于,(d),(a),(用于),),(用于),),),),),(a),(a),(d),(d),(d),(d),(a),),