Deep neural network (DNN) models, including those used in safety-critical domains, need to be thoroughly tested to ensure that they can reliably perform well in different scenarios. In this article, we provide an overview of structural coverage metrics for testing DNN models, including neuron coverage (NC), k-multisection neuron coverage (kMNC), top-k neuron coverage (TKNC), neuron boundary coverage (NBC), strong neuron activation coverage (SNAC) and modified condition/decision coverage (MC/DC). We evaluate the metrics on realistic DNN models used for perception tasks (including LeNet-1, LeNet-4, LeNet-5, and ResNet20) as well as on networks used in autonomy (TaxiNet). We also provide a tool, DNNCov, which can measure the testing coverage for all these metrics. DNNCov outputs an informative coverage report to enable researchers and practitioners to assess the adequacy of DNN testing, compare different coverage measures, and to more conveniently inspect the model's internals during testing.
翻译:深神经网络模型,包括在安全关键领域使用的模型,需要彻底测试,以确保它们在不同的情景下能够可靠地运行良好。在本条中,我们概述了用于测试DNN模型的结构覆盖度量,包括神经覆盖(NC)、K-多科神经覆盖(KMNC)、顶层神经覆盖(TKNC)、神经边界覆盖(NBC)、强大的神经活化覆盖(SNAC)和条件/决定覆盖(MC/DC)。我们评估了用于认知任务的现实的DNN模型(包括LeNet-1、LeNet-4、LeNet-5和ResNet20)以及自主网络(TexiNet)的测量度量度。我们还提供了一种工具,即DNNNCov,可以测量所有这些指标的测试覆盖率。DNNCov输出一份信息覆盖度报告,使研究人员和从业人员能够评估DNN测试是否充分,比较不同的覆盖度,并在测试期间更方便地检查模型的内部。