Although deep feedforward neural networks share some characteristics with the primate visual system, a key distinction is their dynamics. Deep nets typically operate in serial stages wherein each layer completes its computation before processing begins in subsequent layers. In contrast, biological systems have cascaded dynamics: information propagates from neurons at all layers in parallel but transmission occurs gradually over time, leading to speed-accuracy trade offs even in feedforward architectures. We explore the consequences of biologically inspired parallel hardware by constructing cascaded ResNets in which each residual block has propagation delays but all blocks update in parallel in a stateful manner. Because information transmitted through skip connections avoids delays, the functional depth of the architecture increases over time, yielding anytime predictions that improve with internal-processing time. We introduce a temporal-difference training loss that achieves a strictly superior speed-accuracy profile over standard losses and enables the cascaded architecture to outperform state-of-the-art anytime-prediction methods. The cascaded architecture has intriguing properties, including: it classifies typical instances more rapidly than atypical instances; it is more robust to both persistent and transient noise than is a conventional ResNet; and its time-varying output trace provides a signal that can be exploited to improve information processing and inference.
翻译:尽管深饲料向神经网络与灵长视系统具有某些特征,但关键区别在于其动态。深网通常在序列阶段运行,每个层在加工开始之前在随后的层中完成计算。相反,生物系统具有连锁动态:各个层的神经信息平行传播,但传播过程逐渐发生,导致即使是进料向前结构也发生速度-准确性交易。我们通过建造级联ResNet来探索生物启发平行硬件的后果,其中每个残余块都有传播延迟,但所有块都以状态化的方式同步更新。由于通过跳过连接传输的信息可以避免延误,结构的功能深度会随着时间的推移而增加,随着内部处理时间的改善而随时产生预测。我们引入了时间-偏差培训损失,从而使得标准损失的超超超速速度-准确性交易,并使级结构能够超越最先进的时间定位方法。累进结构的特性包括:通过跳过连接传输的信息可以更快地分解典型实例,因此结构结构的功能深度会随着时间而增加,随着时间推移的信号会随着内部处理而不断更新。