Many gradient-based meta-learning methods assume a set of parameters that do not participate in inner-optimization, which can be considered as hyperparameters. Although such hyperparameters can be optimized using the existing gradient-based hyperparameter optimization (HO) methods, they suffer from the following issues. Unrolled differentiation methods do not scale well to high-dimensional hyperparameters or horizon length, Implicit Function Theorem (IFT) based methods are restrictive for online optimization, and short horizon approximations suffer from short horizon bias. In this work, we propose a novel HO method that can overcome these limitations, by approximating the second-order term with knowledge distillation. Specifically, we parameterize a single Jacobian-vector product (JVP) for each HO step and minimize the distance from the true second-order term. Our method allows online optimization and also is scalable to the hyperparameter dimension and the horizon length. We demonstrate the effectiveness of our method on two different meta-learning methods and three benchmark datasets.
翻译:许多基于梯度的元学习方法假定了一系列参数,这些参数不参与内部优化,可被视为超参数。虽然可以利用现有的基于梯度的超参数优化方法优化这些超参数,但它们存在以下问题:无弹性的差别方法不适宜于高维超参数或地平线长度,基于隐形函数理论(IFT)的方法对在线优化有限制性,短期近距离偏差也存在短期偏差。在这项工作中,我们提出一种新的HO方法,通过知识蒸馏接近第二阶术语,可以克服这些局限性。具体地说,我们为每个基于梯度的超参数优化方法设定了单一的Jacobian-VP产品参数,并尽可能缩小与真正的第二阶术语的距离。我们的方法允许在线优化,并且也可以与超参数尺寸和地平线长度相适应。我们用两种不同的元学习方法和三个基准数据集展示了我们的方法的有效性。