Convergence Rates for Multi-classs Logistic Regression Near Minimum

In the current paper we provide constructive estimation of the convergence rate for training a known class of neural networks: the multi-class logistic regression. Despite several decades of successful use, our rigorous results appear new, reflective of the gap between practice and theory of machine learning. Training a neural network is typically done via variations of the gradient descent method. If a minimum of the loss function exists and gradient descent is used as the training method, we provide an expression that relates learning rate to the rate of convergence to the minimum. The method involves an estimate of the condition number of the Hessian of the loss function. We also discuss the existence of a minimum, as it is not automatic that a minimum exists. One method of ensuring convergence is by assigning positive probabiity to every class in the training dataset.