Memory-based Optimization Methods for Model-Agnostic Meta-Learning

Recently, model-agnostic meta-learning (MAML) has garnered tremendous attention. However, the stochastic optimization of MAML is still immature. Existing algorithms for MAML exploit the "episode" idea by sampling some tasks and some data points for each sampled task at each iteration for updating the meta-model. However, they either do not necessarily guarantee convergence with a constant mini-batch size or require processing a large number of tasks at every iteration, which is not viable for continual learning or cross-device federated learning where only a small number of tasks are available per-iteration or per-round. This paper addresses these issues by (i) proposing efficient memory-based stochastic algorithms for MAML with the vanishing convergence error, which only requires sampling a constant number of tasks and a constant number of data samples per-iteration; (ii) proposing communication-efficient distributed memory-based MAML algorithms for personalized federated learning in both the cross-device (with client sampling) and the cross-silo (without client sampling) settings. The theoretical results significantly improve the optimization theory for MAML and the empirical results also corroborate the theory.