Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, time-series, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run using our codebase.
翻译:黑盒机器学习模型现在通常在高风险环境中使用,例如医学诊断,这种模型要求不确定性量化,以避免随之而来的模型失败。 非正式预测是一种方便用户的范例,用于为这类模型的预测创建具有统计性强的不确定性套件/中间体。 关键地说,这些套件在无分配的意义上是有效的:即使没有分发的假设或模型假设,它们也拥有明确的、非不受保护的保障。 人们可以使用任何事先训练的模型,如神经网络,进行符合预测,以简单易懂的模型,保证包含地面真相,并有用户指定的概率,例如90%。这是易于理解的、容易使用的和一般的范例,自然地适用于计算机视觉、自然语言处理、深度加固学习等领域出现的问题。 这个亲手介绍旨在为读者提供对一致预测和相关无分发的不确定性量化技术的工作理解,并使用一个自足的文件。 我们引导读者通过实用的理论和实例,以精确的视角来描述其扩展到复杂的机器学习任务,包括结构化产出、分发、自动语言处理、深入的序列、外推理算法。