This paper addresses the problem of rolling shutter correction in complex nonlinear and dynamic scenes with extreme occlusion. Existing methods suffer from two main drawbacks. Firstly, they face challenges in estimating the accurate correction field due to the uniform velocity assumption, leading to significant image correction errors under complex motion. Secondly, the drastic occlusion in dynamic scenes prevents current solutions from achieving better image quality because of the inherent difficulties in aligning and aggregating multiple frames. To tackle these challenges, we model the curvilinear trajectory of pixels analytically and propose a geometry-based Quadratic Rolling Shutter (QRS) motion solver, which precisely estimates the high-order correction field of individual pixel. Besides, to reconstruct high-quality occlusion frames in dynamic scenes, we present a 3D video architecture that effectively Aligns and Aggregates multi-frame context, namely, RSA^2-Net. We evaluate our method across a broad range of cameras and video sequences, demonstrating its significant superiority. Specifically, our method surpasses the state-of-the-arts by +4.98, +0.77, and +4.33 of PSNR on Carla-RS, Fastec-RS, and BS-RSC datasets, respectively.
翻译:本文解决了在复杂的非线性和动态场景下具有极端遮挡的滚动快门矫正问题。现有方法存在两个主要缺点。首先,由于等速度假设,它们在复杂运动估计准确的矫正场时面临挑战,导致在复杂运动下存在显着的图像矫正误差。其次,动态场景中的剧烈遮挡防止了当前解决方案在对齐和聚合多帧时实现更好的图像质量,这是由于难以对齐和聚合多帧的固有困难。为了解决这些挑战,我们通过几何分析对像素的曲线轨迹进行建模,并提出了一种基于几何的二次滚动快门运动求解器,可以准确估算单个像素的高阶矫正场。此外,为了在动态场景中重构高质量的遮挡帧,我们提出了一种有效的3D视频架构,即RSA^2-Net,能够有效地对齐和聚合多帧环境。我们在广泛的相机和视频序列上评估了我们的方法,证明了它的显着优越性。具体而言,我们的方法在Carla-RS、Fastec-RS和BS-RSC数据集上分别超过了最先进技术的PSNR分别为+4.98、+0.77和+4.33。