Deep learning has proven effective for various application tasks, but its applicability is limited by the reliance on annotated examples. Self-supervised learning has emerged as a promising direction to alleviate the supervision bottleneck, but existing work focuses on leveraging co-occurrences in unlabeled data for task-agnostic representation learning, as exemplified by masked language model pretraining. In this chapter, we explore task-specific self-supervision, which leverages domain knowledge to automatically annotate noisy training examples for end applications, either by introducing labeling functions for annotating individual instances, or by imposing constraints over interdependent label decisions. We first present deep probabilistic logic(DPL), which offers a unifying framework for task-specific self-supervision by composing probabilistic logic with deep learning. DPL represents unknown labels as latent variables and incorporates diverse self-supervision using probabilistic logic to train a deep neural network end-to-end using variational EM. Next, we present self-supervised self-supervision(S4), which adds to DPL the capability to learn new self-supervision automatically. Starting from an initial seed self-supervision, S4 iteratively uses the deep neural network to propose new self supervision. These are either added directly (a form of structured self-training) or verified by a human expert (as in feature-based active learning). Experiments on real-world applications such as biomedical machine reading and various text classification tasks show that task-specific self-supervision can effectively leverage domain expertise and often match the accuracy of supervised methods with a tiny fraction of human effort.
翻译:深层学习已证明对各种应用任务有效,但其适用性因依赖附加说明的例子而受到限制。自我监督学习已成为缓解监督瓶颈的一个大有希望的方向,但现有工作的重点是在未贴标签的数据中利用共同发现来进行任务保密代表性学习,如蒙面语言模型预训所显示的那样。在本章中,我们探索特定任务自我监督,利用域内知识自动为终端应用程序自动注解杂杂杂的培训实例,或者为单个案例加贴标签的标签功能,或者对相互依存的标签决定施加限制。我们首先提出深层次的稳定性逻辑(DPL),通过深层次的学习,为特定任务自我监督提供统一的框架。DPL将未知的标签作为潜在变量,并采用多层次的逻辑,利用基于变异的EM来训练深层神经网络端对端。我们提出自我监督的自我监督(S-4),我们向DPL提供自我监督的自我监督的自我监督应用能力,经常通过深度的服务器自我监督,从初始的自我监督开始,或者直接的自我监督,这些自我监督的自我监督,通过基于基于变异基的自我分析的自我检查的自我检查的自我分析,可以展示的自我分析,这些自我检查的自我测试的自我分析的自我结构的自我分析。