Sequence learning has attracted much research attention from the machine learning community in recent years. In many applications, a sequence learning task is usually associated with multiple temporally correlated auxiliary tasks, which are different in terms of how much input information to use or which future step to predict. For example, (i) in simultaneous machine translation, one can conduct translation under different latency (i.e., how many input words to read/wait before translation); (ii) in stock trend forecasting, one can predict the price of a stock in different future days (e.g., tomorrow, the day after tomorrow). While it is clear that those temporally correlated tasks can help each other, there is a very limited exploration on how to better leverage multiple auxiliary tasks to boost the performance of the main task. In this work, we introduce a learnable scheduler to sequence learning, which can adaptively select auxiliary tasks for training depending on the model status and the current training data. The scheduler and the model for the main task are jointly trained through bi-level optimization. Experiments show that our method significantly improves the performance of simultaneous machine translation and stock trend forecasting.
翻译:近年来,序列学习吸引了机器学习界的大量研究关注。在许多应用中,序列学习任务通常与多个时间相关联的辅助任务相关,这些任务在使用多少投入信息或预测未来步骤方面各不相同。例如,(一)在同时机翻译方面,人们可以在不同的时间间隔下进行翻译(即,有多少输入词需要读/等待翻译);(二)在存量趋势预测方面,人们可以预测未来不同日期(例如,明天,后天)的存量价格。虽然这些时间相关的任务显然可以相互帮助,但在如何更好地利用多个辅助任务来提高主要任务绩效方面,探索非常有限。在这项工作中,我们引入了可学习的排序表,根据模型状态和当前培训数据,可以调整选择培训的辅助任务。主要任务的调度器和模型通过双级优化共同培训。实验表明,我们的方法大大改进了同时机器翻译和库存趋势预测的绩效。