In the wake of the explosive growth in smartphones and cyberphysical systems, there has been an accelerating shift in how data is generated away from centralised data towards on-device generated data. In response, machine learning algorithms are being adapted to run locally on board, potentially hardware limited, devices to improve user privacy, reduce latency and be more energy efficient. However, our understanding of how these device orientated algorithms behave and should be trained is still fairly limited. To address this issue, a method to automatically synthesize reduced-order neural networks (having fewer neurons) approximating the input/output mapping of a larger one is introduced. The reduced-order neural network's weights and biases are generated from a convex semi-definite programme that minimises the worst-case approximation error with respect to the larger network. Worst-case bounds for this approximation error are obtained and the approach can be applied to a wide variety of neural networks architectures. What differentiates the proposed approach to existing methods for generating small neural networks, e.g. pruning, is the inclusion of the worst-case approximation error directly within the training cost function, which should add robustness. Numerical examples highlight the potential of the proposed approach. The overriding goal of this paper is to generalise recent results in the robustness analysis of neural networks to a robust synthesis problem for their weights and biases.
翻译:智能话机和网络物理系统爆炸性增长后,数据从集中数据生成方式从集中数据向输入/输出绘图转换为在线生成数据的方式正在加速转变。作为回应,正在对机器学习算法进行调整,以适应本地操作,可能硬件有限,使用机器学习算法来改善用户隐私,减少延时率,提高能效;然而,我们对这些设备定向算法如何表现和应接受培训的理解仍然相当有限。为解决这一问题,一种自动合成减少的神经神经网络(减少神经系统)与较大神经网络的输入/输出映射相匹配的方法被引入了。减序神经网络的权重和偏向性正在从一个配置的半确定性方案中产生,该半确定程序可以最大限度地减少与更大网络有关的最坏的近似误差。我们获得了这种近似性算算法的最坏的界限,该方法可以应用于各种各样的神经网络结构。对创建小型神经网络的现有方法的拟议方法有区别,例如运行线,这是将最坏性神经性神经性网络的权重性分析结果纳入到最强性的文件中。