The application of Deep Neural Networks (DNNs) to a broad variety of tasks demands methods for coping with the complex and opaque nature of these architectures. The analysis of performance can be pursued in two ways. On one side, model interpretation techniques aim at "opening the box" to assess the relationship between the input, the inner layers, and the output. For example, saliency and attention models exploit knowledge of the architecture to capture the essential regions of the input that have the most impact on the inference process and output. On the other hand, models can be analysed as "black boxes", e.g., by associating the input samples with extra annotations that do not contribute to model training but can be exploited for characterizing the model response. Such performance-driven meta-annotations enable the detailed characterization of performance metrics and errors and help scientists identify the features of the input responsible for prediction failures and focus their model improvement efforts. This paper presents a structured survey of the tools that support the "black box" analysis of DNNs and discusses the gaps in the current proposals and the relevant future directions in this research field.
翻译:深神经网络(DNNs)应用于各种各样的任务,要求用多种方法来应付这些结构的复杂和不透明性质。可以用两种方式分析性能。一方面,模型解释技术旨在“打开盒子”以评估输入、内层和产出之间的关系。例如,突出和关注模型利用结构知识来捕捉对推断过程和产出影响最大的投入的基本区域。另一方面,模型可以作为“黑盒”分析,例如,将输入样本与无助于示范培训但可用于描述模型响应特性的额外说明联系起来。这种以性能为驱动的元说明有助于详细描述性能指标和错误,并帮助科学家确定负责预测失败的投入的特点,并集中开展模型改进工作。本文件对支持DNPs“黑盒”分析的工具进行分阶段调查,并讨论当前建议中的差距和本研究领域的有关未来方向。