Uncertainty estimation is important for ensuring safety and robustness of AI systems. While most research in the area has focused on un-structured prediction tasks, limited work has investigated general uncertainty estimation approaches for structured prediction. Thus, this work aims to investigate uncertainty estimation for autoregressive structured prediction tasks within a single unified and interpretable probabilistic ensemble-based framework. We consider: uncertainty estimation for sequence data at the token-level and complete sequence-level; interpretations for, and applications of, various measures of uncertainty; and discuss both the theoretical and practical challenges associated with obtaining them. This work also provides baselines for token-level and sequence-level error detection, and sequence-level out-of-domain input detection on the WMT'14 English-French and WMT'17 English-German translation and LibriSpeech speech recognition datasets.
翻译:不确定性的估算对于确保AI系统的安全和稳健性十分重要。虽然这一领域的大多数研究侧重于非结构化的预测任务,但有限的工作调查了结构化预测的一般不确定性估算方法,因此,这项工作的目的是在一个单一的统一和可解释的共性框架范围内,调查自动递减结构化预测任务的不确定性估算。我们考虑:象征性和完整序列级别的序列数据的不确定性估算;各种不确定性计量的解释和应用;讨论与获取这些数据有关的理论和实际挑战。这项工作还为在WMT'14英语-法语和WMT'17英语-德语翻译和LibriSpeech语音识别数据集中进行象征性和序列级误差检测以及序列级外输入检测提供了基线。