In contrast to traditional weight optimization in a continuous space, we demonstrate the existence of effective random networks whose weights are never updated. By selecting a weight among a fixed set of random values for each individual connection, our method uncovers combinations of random weights that match the performance of traditionally-trained networks of the same capacity. We refer to our networks as "slot machines" where each reel (connection) contains a fixed set of symbols (random values). Our backpropagation algorithm "spins" the reels to seek "winning" combinations, i.e., selections of random weight values that minimize the given loss. Quite surprisingly, we find that allocating just a few random values to each connection (e.g., 8 values per connection) yields highly competitive combinations despite being dramatically more constrained compared to traditionally learned weights. Moreover, finetuning these combinations often improves performance over the trained baselines. A randomly initialized VGG-19 with 8 values per connection contains a combination that achieves 91% test accuracy on CIFAR-10. Our method also achieves an impressive performance of 98.2% on MNIST for neural networks containing only random weights.
翻译:与连续空间的传统重力优化相比, 我们展示了有效随机网络的存在, 其重量从未更新。 通过选择每个连接的固定随机值中的权重, 我们的方法发现随机权重的组合, 与传统上受过训练的相同容量网络的性能相匹配。 我们称我们的网络为“ 绘图机 ”, 其中每个连线( 连接) 包含固定的符号( 随机值 ) 。 我们的回向回调算法“ 螺旋”, 寻找“ 结对” 组合, 即选择随机权重值, 最大限度地减少给定的损失。 非常令人惊讶的是, 我们的方法发现, 仅仅为每个连接分配少数随机值( 例如, 每连接8 个值), 就能产生高度竞争性的组合, 尽管与传统上学到的重量相比, 限制要大得多。 此外, 微调这些组合往往能提高经过训练的基线的性能。 我们随机初始的 VGGG-19 和8 连接值包含一个组合, 使得CIFAR- 10 的测试精准度达到91%。 我们的方法还实现了仅包含随机重量网络的98.2% MNMISTISISTISCIST 。