Constructing useful representations across a large number of tasks is a key requirement for sample-efficient intelligent systems. A traditional idea in multitask learning (MTL) is building a shared representation across tasks which can then be adapted to new tasks by tuning last layers. A desirable refinement of using a shared one-fits-all representation is to construct task-specific representations. To this end, recent PathNet/muNet architectures represent individual tasks as pathways within a larger supernet. The subnetworks induced by pathways can be viewed as task-specific representations that are composition of modules within supernet's computation graph. This work explores the pathways proposal from the lens of statistical learning: We first develop novel generalization bounds for empirical risk minimization problems learning multiple tasks over multiple paths (Multipath MTL). In conjunction, we formalize the benefits of resulting multipath representation when adapting to new downstream tasks. Our bounds are expressed in terms of Gaussian complexity, lead to tangible guarantees for the class of linear representations, and provide novel insights into the quality and benefits of a multipath representation. When computation graph is a tree, Multipath MTL hierarchically clusters the tasks and builds cluster-specific representations. We provide further discussion and experiments for hierarchical MTL and rigorously identify the conditions under which Multipath MTL is provably superior to traditional MTL approaches with shallow supernets.
翻译:多任务学习(MTL)中的一种传统理念是,通过对最后层进行调适,建立跨任务的共同代表制,从而适应新的任务。使用共享的“一刀切”代表制的可取改进是,构建具体任务的代表制。为此,最近的PathNet/muNet结构将单个任务作为大超级网的路径。路径引发的子网络可被视为任务特定代表制,由超级网计算图中的模块构成。这项工作从统计学习的角度探讨路径建议:我们首先为经验风险最小化问题制定新的通用范围,通过多途径学习多重任务(MTL)。我们一起,在适应新的下游任务时,将由此产生的多路代表制的好处正式化。我们的界限以高频复杂度表示,为直线表达的类别提供切实的保证,并为多路径代表制的质量和利益提供新的洞见。当进一步计算图表是一棵树,多方向MTL多方向,通过高层次、高层次、高层次、高层次、高层次、高层次、高层次的分组代表制,我们通过高层次、高层次、高层次、高层次、高层次、高层次、高层次、高层次MTMTL任务和多面图展示。</s>