Traditional supervised learning mostly works on individual tasks and requires training on a large set of task-specific examples. This paradigm seriously hinders the development of task generalization since preparing a task-specific example set is costly. To build a system that can quickly and easily generalize to new tasks, task instructions have been adopted as an emerging trend of supervision recently. These instructions give the model the definition of the task and allow the model to output the appropriate answer based on the instructions and inputs. However, task instructions are often expressed in different forms, which can be interpreted from two threads: first, some instructions are short sentences and are pretrained language model (PLM) oriented, such as prompts, while other instructions are paragraphs and are human-oriented, such as those in Amazon MTurk; second, different end-users very likely explain the same task with instructions of different textual expressions. A robust system for task generalization should be able to handle any new tasks regardless of the variability of instructions. However, the system robustness in dealing with instruction-driven task generalization is still unexplored. This work investigates the system robustness when the instructions of new tasks are (i) maliciously manipulated, (ii) paraphrased, or (iii) from different levels of conciseness. To our knowledge, this is the first work that systematically studies how robust a PLM is when it is supervised by instructions with different factors of variability.
翻译:传统监督的学习大多针对个别任务,需要就大量具体任务实例进行培训。这种范式严重阻碍了任务一般化的发展,因为编写具体任务范例的费用很高。要建立一个能够迅速和方便地概括新任务的系统,最近已经将任务指示作为新出现的监督趋势。这些指示给模式规定了任务的定义,并使模式能够根据指示和投入产生适当的答案。然而,任务指示往往以不同的形式表述,可以从两个线索来解释:第一,有些指示是短句,是预先训练的语言模式(PLM),如提示,而其他指示则是段落,面向人;第二,不同的最终用户很可能以不同文本表达方式的指示解释同样的任务。一个强有力的任务一般化系统应该能够处理任何新的任务,而不管指示的变异性如何。然而,处理指导驱动任务一般化的系统是否稳健,仍然无法解释。当新任务指示的指令是(i)与新指令的精细性、多变异性研究是系统化的(i)时,这项工作是系统化的。