Diffusion Mechanism in Neural Network: Theory and Applications

Diffusion is the movement of molecules from a region of higher concentration to a region of lower concentration. It can be used to describe the interactions among data points. In many machine learning problems including transductive semi-supervised learning and few-shot learning, the relationship between labeled and unlabeled data points is a critical component for high classification accuracy. In this paper, inspired by the convection-diffusion ODE, we propose a novel diffusion residual network (Diff-ResNet) to introduce diffusion mechanism into neural networks internally. Under the structured data assumption, it is proved that the diffusion mechanism can increase the distance-diameter ratio that improves the separability of inter-class points and reduces the distance among local intra-class points. This property can be easily adopted by the residual networks for constructing the separable hyperplanes. Extensive experiments of synthetic binary classification, semi-supervised graph node classification and few-shot image classification in various datasets validate the effectiveness of the proposed diffusion mechanism.