Efficient and provably convergent randomized greedy algorithms for neural network optimization

Greedy algorithms have been successfully analyzed and applied in training neural networks for solving variational problems, ensuring guaranteed convergence orders. However, their practical applicability is limited due to the subproblems, which often require an exhaustive search over a discrete dictionary and incur significant computational costs. This limitation becomes critical, especially in high-dimensional problems. In this paper, we propose a more practical approach of randomly discretizing the dictionary at each iteration of the greedy algorithm. We quantify the required size of the randomized discrete dictionary and prove that, with high probability, the proposed algorithm realizes a weak greedy algorithm, achieving optimal convergence orders. Through numerous numerical experiments, we demonstrate the advantage of using randomized discrete dictionaries over a deterministic one by showing orders of magnitude reductions in the size of the discrete dictionary, particularly in higher dimensions.