Activation functions (AFs) are an important part of the design of neural networks (NNs), and their choice plays a predominant role in the performance of a NN. In this work, we are particularly interested in the estimation of flexible activation functions using tensor-based solutions, where the AFs are expressed as a weighted sum of predefined basis functions. To do so, we propose a new learning algorithm which solves a constrained coupled matrix-tensor factorization (CMTF) problem. This technique fuses the first and zeroth order information of the NN, where the first-order information is contained in a Jacobian tensor, following a constrained canonical polyadic decomposition (CPD). The proposed algorithm can handle different decomposition bases. The goal of this method is to compress large pretrained NN models, by replacing subnetworks, {\em i.e.,} one or multiple layers of the original network, by a new flexible layer. The approach is applied to a pretrained convolutional neural network (CNN) used for character classification.
翻译:激活功能(AFs)是神经网络设计的一个重要部分,它们的选择在NN的运行中起着主要作用。在这项工作中,我们特别感兴趣的是使用基于压力的解决方案来估计灵活的激活功能,因为AF是以预先界定的基函数的加权和总和表示的。为此,我们提议一种新的学习算法,解决受限制的组合矩阵-加速因子化(CMTF)问题。这种技术将NN的一级和零级信息结合在一起,在限制的卡门聚变分解(CPD)后,第一个级信息包含在Jacobian Exmor中。提议的算法可以处理不同的分解基。这种方法的目的是通过新的灵活层替换子网络,即(em) 或(i.e.}) 将原网络的一或多层压缩成。该方法适用于用于性格分类的预先训练的革命性神经网络(CNN)。