Deep neural networks (DNNs) play an increasingly important role in various computer systems. In order to create these networks, engineers typically specify a desired topology, and then use an automated training algorithm to select the network's weights. While training algorithms have been studied extensively and are well understood, the selection of topology remains a form of art, and can often result in networks that are unnecessarily large - and consequently are incompatible with end devices that have limited memory, battery or computational power. Here, we propose to address this challenge by harnessing recent advances in DNN verification. We present a framework and a methodology for discovering redundancies in DNNs - i.e., for finding neurons that are not needed, and can be removed in order to reduce the size of the DNN. By using sound verification techniques, we can formally guarantee that our simplified network is equivalent to the original, either completely, or up to a prescribed tolerance. Further, we show how to combine our technique with slicing, which results in a family of very small DNNs, which are together equivalent to the original. Our approach can produce DNNs that are significantly smaller than the original, rendering them suitable for deployment on additional kinds of systems, and even more amenable to subsequent formal verification. We provide a proof-of-concept implementation of our approach, and use it to evaluate our techniques on several real-world DNNs.
翻译:深心神经网络(DNNS)在各种计算机系统中发挥着越来越重要的作用。为了创建这些网络,工程师们通常会指定一个理想的地形学,然后使用自动培训算法来选择网络的重量。虽然培训算法已经进行了广泛的研究,并且人们非常理解,但选择地形学仍然是一种艺术形式,而且往往会导致不必要的庞大网络,因此,与记忆、电池或计算能力有限的终端设备不相容。我们在这里建议利用DNNN核查的最新进展来应对这一挑战。我们提出了一个在DNNNS中发现冗余的框架和方法,即寻找不需要的神经元,并且可以去除,以便缩小DNNN的体大小。我们可以通过使用健全的核查技术,正式保证我们简化的网络相当于原始的,或者完全的,或者达到规定的容忍度。此外,我们展示了如何将我们的技术与剪切装置结合起来,从而形成一个非常小的DNNNS的大家庭,这相当于原始的大家庭。我们的方法可以产生一个真正的DNNS方法,甚至更小的部署方式, 用来使DNNF系统更适合世界的系统。