This paper proposes the first real-world rolling shutter (RS) correction dataset, BS-RSC, and a corresponding model to correct the RS frames in a distorted video. Mobile devices in the consumer market with CMOS-based sensors for video capture often result in rolling shutter effects when relative movements occur during the video acquisition process, calling for RS effect removal techniques. However, current state-of-the-art RS correction methods often fail to remove RS effects in real scenarios since the motions are various and hard to model. To address this issue, we propose a real-world RS correction dataset BS-RSC. Real distorted videos with corresponding ground truth are recorded simultaneously via a well-designed beam-splitter-based acquisition system. BS-RSC contains various motions of both camera and objects in dynamic scenes. Further, an RS correction model with adaptive warping is proposed. Our model can warp the learned RS features into global shutter counterparts adaptively with predicted multiple displacement fields. These warped features are aggregated and then reconstructed into high-quality global shutter frames in a coarse-to-fine strategy. Experimental results demonstrate the effectiveness of the proposed method, and our dataset can improve the model's ability to remove the RS effects in the real world.
翻译:本文提出了第一个真实世界滚动百叶窗(RS)校正数据集(BS-RSC),以及一个在扭曲的视频中校正RS框架的相应模型。消费市场中带有基于CMOS的视频摄取传感器的流动设备,在视频获取过程中发生相对移动时,往往会产生滚动百叶窗效应,要求采用RS效应清除技术。然而,目前最先进的RS校正方法往往无法在真实情景中消除RS效应,因为这些动作多种多样,难以建模。为了解决这个问题,我们提出了真实世界RS校正数据集(BS-RSC)。真实的扭曲视频与相应的地面真相同时通过精心设计的光谱采集系统进行记录。BS-RSC包含动态场景中照相机和物体的各种运动。此外,还提出了具有适应扭曲效果的RS校正模型。我们的模型可以将所学到的RS特性转换成适应预测的多个迁移场的全球百叶窗相。这些扭曲的特征被汇总,然后重建成高品质的全球封闭框。在粗糙的模型战略中,通过实验结果显示我们所提议方法的实效,可以改变世界。