Multilevel models using logistic regression (MLogRM) and random forest models (RFM) are increasingly deployed in industry for the purpose of binary classification. The European Commission's proposed Artificial Intelligence Act (AIA) necessitates, under certain conditions, that application of such models is fair, transparent, and ethical, which consequently implies technical assessment of these models. This paper proposes and demonstrates an audit framework for technical assessment of RFMs and MLogRMs by focussing on model-, discrimination-, and transparency & explainability-related aspects. To measure these aspects 20 KPIs are proposed, which are paired to a traffic light risk assessment method. An open-source dataset is used to train a RFM and a MLogRM model and these KPIs are computed and compared with the traffic lights. The performance of popular explainability methods such as kernel- and tree-SHAP are assessed. The framework is expected to assist regulatory bodies in performing conformity assessments of binary classifiers and also benefits providers and users deploying such AI-systems to comply with the AIA.
翻译:利用后勤回归(MLogRM)和随机森林模型(RFM)的多层次模型越来越多地用于工业,以进行二进制分类。欧盟委员会拟议的《人工情报法》(AIA)要求在某些条件下应用这些模型是公平、透明和合乎道德的,因此意味着对这些模型进行技术评估。本文件提出并展示了对RFM和MLogRM技术评估的审计框架,重点是模型、歧视、透明度和可解释性方面。为衡量这些方面,提出了20个KPI,这些方面与交通轻风险评估方法相配。使用开放源数据集来培训RFM和MLogRM模型,并与交通灯进行比较。这些KPIS进行计算和比较。评估了诸如内核和树木-SHAP等可解释性方法的性能。预计该框架将协助管理机构对二进制分类器进行合规评估,同时也为使用这种AI系统以遵守AIA提供和用户。