As a bio-inspired sensor with high temporal resolution, the spiking camera has an enormous potential in real applications, especially for motion estimation in high-speed scenes. However, frame-based and event-based methods are not well suited to spike streams from the spiking camera due to the different data modalities. To this end, we present, SCFlow, a tailored deep learning pipeline to estimate optical flow in high-speed scenes from spike streams. Importantly, a novel input representation is introduced which can adaptively remove the motion blur in spike streams according to the prior motion. Further, for training SCFlow, we synthesize two sets of optical flow data for the spiking camera, SPIkingly Flying Things and Photo-realistic High-speed Motion, denoted as SPIFT and PHM respectively, corresponding to random high-speed and well-designed scenes. Experimental results show that the SCFlow can predict optical flow from spike streams in different high-speed scenes. Moreover, SCFlow shows promising generalization on \real spike streams. All codes and constructed datasets will be released after publication.
翻译:作为具有高时间分辨率的生物感应器,冲刺相机在实际应用方面有着巨大的潜力,特别是在高速场景的运动估计方面。然而,由于数据模式不同,基于框架和事件的方法并不完全适合冲刺摄影机的冲刺流。为此,我们介绍SCFlow,这是一条量身定制的深层学习管道,用于估计冲刺流高速场景的光学流动。重要的是,引入了一种新的输入代表,能够根据先前的动作适应性地消除冲刺流中的模糊移动。此外,为了培训SCFlow,我们为冲刺摄影机合成了两套光学流数据,即SPIFT和PHMT,分别称为SPIFT和PHM,分别称为SPIF和PHM。实验结果显示,SCFlow可以预测不同高速场景的冲刺流的光学流。此外,SCFLlow展示了对冲刺流有希望的普遍化。所有代码和构建的数据集将在出版后发布。