背景:只专注于单个模型可能会忽略一些相关任务中可能提升目标任务的潜在信息,通过进行一定程度的共享不同任务之间的参数,可能会使原任务泛化更好。广义的讲,只要loss有多个就算MTL,一些别名(joint learning,learning to learn,learning with auxiliary task)
20多岁的hard parameter shareing还是很流行,目前热点learning what to learn也很有价值,我们对tasks的理解(similarity, relationship, hierrarchy, benefit for MTL) 还是很有限的,希望以后有重大发展吧。
可研究方向
learning what to share
measurement for similarity of tasks
using task uncertainty
引入异步任务(特征学习任务),采用交替迭代训练
学习抽象子任务;学习任务结构(类似强化里面的hierarchy learning)
参数学习辅助任务
More...
备注:本文学习资料主要来自 _An Overview of Multi-Task Learning in Deep Neural Networks,https://arxiv.org/abs/1706.05098
Reference
[1] A Bayesian/information theoretic model of learning to learn via multiple task sampling. http://link.springer.com/article/10.1023/A:1007327622663
[2] Learning from hints in neural networks. Journal of Complexity https://doi.org/10.1016/0885-064X(90)90006-Y
[4] Model selection and estimation in regression with grouped variables
[5] Taking Advantage of Sparsity in Multi-Task Learninghttp://arxiv.org/pdf/0903.1468
[6] A Dirty Model for Multi-task Learning. Advances in Neural Information Processing Systems https://papers.nips.cc/paper/4125-a-dirty-model-for-multi-task-learning.pdf
[9] Discovering Structure in Multiple Learning Tasks: The TC Algorithm http://scholar.google.com/scholar?cluster=956054018507723832&hl=en
[10] A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data
[11] Empirical Bayes for Learning to Learn
[12] Learning to learn with the informative vector machine https://doi.org/10.1145/1015330.1015382
[13] Task Clustering and Gating for Bayesian Multitask Learning https://doi.org/10.1162/153244304322765658
[14] Multi-Task Learning for Classification with Dirichlet Process Priors
[15] Bayesian multitask learning with latent hierarchies http://dl.acm.org.sci-hub.io/citation.cfm?id=1795131
[16] Linear Algorithms for Online Multitask Classification
[17] Learning with whom to share in multi-task feature learning
[18] Learning Task Grouping and Overlap in Multi-task Learning
[19] Learning Multiple Tasks Using Shared Hypotheses
[20] Learning Multiple Tasks with Deep Relationship Networks http://arxiv.org/abs/1506.02117
[21] Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification http://arxiv.org/abs/1611.05377
[22] Cross-stitch Networks for Multi-task Learning https://doi.org/10.1109/CVPR.2016.433
[23] Deep multi-task learning with low level tasks supervised at lower layers
[24] A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks http://arxiv.org/abs/1611.01587
[25] Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics http://arxiv.org/abs/1705.07115
[26] Deep Multi-task Representation Learning: A Tensor Factorisation Approach https://doi.org/10.1002/joe.20070
[27] Sluice networks: Learning what to share between loosely related tasks http://arxiv.org/abs/1705.08142
[28] Exploiting task relatedness for multiple task learning. Learning Theory and Kernel Machines https://doi.org/10.1007/978-3-540-45167-9_41
[29] When is multitask learning effective? Multitask learning for semantic sequence prediction under varying data conditions http://arxiv.org/abs/1612.02251
[30] Identifying beneficial task relations for multi-task learning in deep neural networks http://arxiv.org/abs/1702.08303
[31] Multitask learning using uncertainty to weigh losses for scene geometry and senantics