Modern machine learning algorithms are capable of providing remarkably accurate point-predictions; however, questions remain about their statistical reliability. Unlike conventional machine learning methods, conformal prediction algorithms return confidence sets (i.e., set-valued predictions) that correspond to a given significance level. Moreover, these confidence sets are valid in the sense that they guarantee finite sample control over type 1 error probabilities, allowing the practitioner to choose an acceptable error rate. In our paper, we propose inductive conformal prediction (ICP) algorithms for the tasks of text infilling and part-of-speech (POS) prediction for natural language data. We construct new conformal prediction-enhanced bidirectional encoder representations from transformers (BERT) and bidirectional long short-term memory (BiLSTM) algorithms for POS tagging and a new conformal prediction-enhanced BERT algorithm for text infilling. We analyze the performance of the algorithms in simulations using the Brown Corpus, which contains over 57,000 sentences. Our results demonstrate that the ICP algorithms are able to produce valid set-valued predictions that are small enough to be applicable in real-world applications. We also provide a real data example for how our proposed set-valued predictions can improve machine generated audio transcriptions.
翻译:现代机器学习算法能够提供非常准确的点值; 但是,关于统计可靠性的问题仍然存在。 与常规的机器学习方法不同, 符合的预测算法返回了符合某一重要程度的一套信心( 定值预测 ) 。 此外, 这些信任组是有效的, 因为它们保证对第1类误差概率进行有限的抽样控制, 让执业者能够选择可接受的误差率 。 在我们的文件中, 我们提议对自然语言数据的文本填充和部分语音( POS)预测任务进行感应一致预测( ICP ) 算法 。 我们从变压器(BERT) 和双向长期短期内存(BILSTM) 算法中构建了新的符合的预测- 增强双向双向的双向编码算法 。 我们用布朗公司( Prown Corpus) 算法分析模拟算法的绩效, 包含57 000多条小句子。 我们的结果表明, 比较方案算法能够充分改进真实的定值预测。 我们的定值数据也能够提供正确的定值, 我们的定值的定值数据, 我们的定值的定值的定值的定值的定值可以提供正确的定值, 的定值数据能的模型的模型的模型的模型的模型的模型的模型可以提供出如何。