When trying to fit a deep neural network (DNN) to a $G$-invariant target function with $G$ a group, it only makes sense to constrain the DNN to be $G$-invariant as well. However, there can be many different ways to do this, thus raising the problem of "$G$-invariant neural architecture design": What is the optimal $G$-invariant architecture for a given problem? Before we can consider the optimization problem itself, we must understand the search space, the architectures in it, and how they relate to one another. In this paper, we take a first step towards this goal; we prove a theorem that gives a classification of all $G$-invariant single-hidden-layer or "shallow" neural network ($G$-SNN) architectures with ReLU activation for any finite orthogonal group $G$, and we prove a second theorem that characterizes the inclusion maps or "network morphisms" between the architectures that can be leveraged during neural architecture search (NAS). The proof is based on a correspondence of every $G$-SNN to a signed permutation representation of $G$ acting on the hidden neurons; the classification is equivalently given in terms of the first cohomology classes of $G$, thus admitting a topological interpretation. The $G$-SNN architectures corresponding to nontrivial cohomology classes have, to our knowledge, never been explicitly identified in the literature previously. Using a code implementation, we enumerate the $G$-SNN architectures for some example groups $G$ and visualize their structure. Finally, we prove that architectures corresponding to inequivalent cohomology classes coincide in function space only when their weight matrices are zero, and we discuss the implications of this for NAS.
翻译:当试图将一个深层神经网络(DNN) 与一个 G$- 变量目标函数相匹配时, 一个组用 G$ 来匹配一个 G$- 变化中的目标函数时, 将 DNN 限制为 G$ 也是 G$- 变化中。 但是, 这样做可以有许多不同的方式, 从而提出了“ $ G$- 变化中神经结构设计 ” 的问题: 当我们考虑一个特定问题时, 最理想的 $- 变化中的结构是什么? 在我们考虑优化问题本身之前, 我们必须理解搜索空间、 其中的结构以及它们彼此的关系。 在本文中, 我们迈出了第一步, 我们为这个目标而迈出了第一步; 我们证明了一个理论, 将所有G$- 变化中的 美元 单项- G- 神经网络设计($- SNNNNN) 结构分类( $ ) 。 因此, 在给定值中, 其内部结构( NON- 值中, ) 显示一个对应的代码, 当我们第一次使用时, 我们的直系直系的直系的直系的直系的直线图。