Neural ordinary differential equations (ODEs) have been attracting increasing attention in various research domains recently. There have been some works studying optimization issues and approximation capabilities of neural ODEs, but their robustness is still yet unclear. In this work, we fill this important gap by exploring robustness properties of neural ODEs both empirically and theoretically. We first present an empirical study on the robustness of the neural ODE-based networks (ODENets) by exposing them to inputs with various types of perturbations and subsequently investigating the changes of the corresponding outputs. In contrast to conventional convolutional neural networks (CNNs), we find that the ODENets are more robust against both random Gaussian perturbations and adversarial attack examples. We then provide an insightful understanding of this phenomenon by exploiting a certain desirable property of the flow of a continuous-time ODE, namely that integral curves are non-intersecting. Our work suggests that, due to their intrinsic robustness, it is promising to use neural ODEs as a basic block for building robust deep network models. To further enhance the robustness of vanilla neural ODEs, we propose the time-invariant steady neural ODE (TisODE), which regularizes the flow on perturbed data via the time-invariant property and the imposition of a steady-state constraint. We show that the TisODE method outperforms vanilla neural ODEs and also can work in conjunction with other state-of-the-art architectural methods to build more robust deep networks. \url{https://github.com/HanshuYAN/TisODE}
翻译:最近,在各种研究领域,神经普通差异方程式(ODEs)吸引了越来越多的注意力。有些研究研究优化问题和神经元代码近似能力的工作,但还不清楚。在这项工作中,我们通过在实验和理论上探索神经值代码的稳健性特性来填补这一重要差距。我们首先对神经值代码网络(ODENets)的稳健性能进行了实证研究,通过不同类型的扰动,然后调查相应产出的变化。与传统的神经神经网络(CNNs)相比,我们发现ODENets对随机的Gaussian扰动和对抗性攻击实例都更为强大。我们随后通过利用连续时间值代码网络(ODENets)流的某些理想性能来提供对这一现象的深刻理解。我们的工作表明,由于其内在的强力性能,我们有望使用神经值模型作为构建更强的深度网络模型(CNNNNNNERs)的基本基石。为了进一步增强动态神经值网络的稳健性运行力,我们还可以通过恒定的神经值模型来显示稳定的神经值内值。