Deep Neural Network-based systems are now the state-of-the-art in many robotics tasks, but their application in safety-critical domains remains dangerous without formal guarantees on network robustness. Small perturbations to sensor inputs (from noise or adversarial examples) are often enough to change network-based decisions, which was recently shown to cause an autonomous vehicle to swerve into another lane. In light of these dangers, numerous algorithms have been developed as defensive mechanisms from these adversarial inputs, some of which provide formal robustness guarantees or certificates. This work leverages research on certified adversarial robustness to develop an online certifiably robust for deep reinforcement learning algorithms. The proposed defense computes guaranteed lower bounds on state-action values during execution to identify and choose a robust action under a worst-case deviation in input space due to possible adversaries or noise. Moreover, the resulting policy comes with a certificate of solution quality, even though the true state and optimal action are unknown to the certifier due to the perturbations. The approach is demonstrated on a Deep Q-Network policy and is shown to increase robustness to noise and adversaries in pedestrian collision avoidance scenarios and a classic control task. This work extends one of our prior works with new performance guarantees, extensions to other RL algorithms, expanded results aggregated across more scenarios, an extension into scenarios with adversarial behavior, comparisons with a more computationally expensive method, and visualizations that provide intuition about the robustness algorithm.
翻译:以深心网络为基础的系统现在已成为许多机器人任务中最先进的先进技术,但在安全关键领域的应用仍然十分危险,没有正式的网络稳健度保障。对传感器投入(来自噪音或对抗性实例)的小扰动往往足以改变基于网络的决定,最近显示,这导致一个自主的车辆向另一条路倾斜。鉴于这些危险,许多算法已经发展成为这些对抗性投入的防御机制,其中一些是正式的稳健保证或证书。这项工作利用了对经认证的对抗性强力的研究,为深度强化学习算法开发了可证实的在线稳健度。拟议的国防计算保证了执行期间对州行动值的较低限制,从而在可能敌对或噪音造成的投入空间最差的情况下确定和选择了强有力的行动。此外,由此产生的政策带有解决方案质量的证明,尽管由于扭曲性比较,真正的状态和最佳行动是未知的。这种方法在深度网络化政策上展示了一种可验证的稳健的稳健性强度,并展示了深度强化的强化的强化强化强化强化强化强化的强化的强化的强化的强化学习算算算算算法。 拟议的防御性假设将了我们更具有更稳健的升级的升级的升级的逻辑,从而扩大了了在避免碰撞任务前的逻辑上的风险推延延延后的工作。