To systematically validate the safe behavior of automated vehicles (AV), the aim of scenario-based testing is to cluster the infinite situations an AV might encounter into a finite set of functional scenarios. Every functional scenario, however, can still manifest itself in a vast amount of variations. Thus, metamodels are often used to perform analyses or to select specific variations for examination. However, despite the safety criticalness of AV testing, metamodels are usually seen as a part of an overall approach, and their predictions are not further examined. In this paper, we analyze the predictive performance of Gaussian processes (GP), deep Gaussian processes, extra-trees (ET), and Bayesian neural networks (BNN), considering four scenarios with 5 to 20 inputs. Building on this, we introduce and evaluate an iterative approach to efficiently select test cases. Our results show that regarding predictive performance, the appropriate selection of test cases is more important than the choice of metamodels. While their great flexibility allows BNNs to benefit from large amounts of data and to model even the most complex scenarios, less flexible models like GPs can convince with higher reliability. This implies that relevant test cases have to be explored using scalable virtual environments and flexible models so that more realistic test environments and more trustworthy models can be used for targeted testing and validation.
翻译:为了系统地验证自动飞行器的安全行为,基于假设情况的测试的目的是将AV可能遇到的无限情况集中到一套有限的功能假设中,但每种功能假设都仍然可以表现为巨大的变异。因此,元模型常常用来进行分析或选择具体的变异,供检查。然而,尽管AV测试具有安全重要性,但元模型通常被视为整体方法的一部分,其预测没有进一步审查。在本文件中,我们分析Gaussian进程(GP)、深高山进程、外部树木(ET)和巴耶斯神经网络(BNN)的预测性业绩,考虑有5至20个投入的四种假想。在此基础上,我们引入并评价一种迭接方法,以便有效地选择测试案例。我们的结果显示,在预测性表现方面,适当选择测试案例比选择元模型更为重要。虽然这些模型的巨大灵活性使BNNP从大量数据中受益,甚至模拟最复杂的假想,但较不灵活的模型,例如GPPs等模型能够以更可靠的方式说服人信服,而且能够更可靠地用更可靠的虚拟的测试环境进行更精确的测试。