How to properly model the inter-frame relation within the video sequence is an important but unsolved challenge for video restoration (VR). In this work, we propose an unsupervised flow-aligned sequence-to-sequence model (S2SVR) to address this problem. On the one hand, the sequence-to-sequence model, which has proven capable of sequence modeling in the field of natural language processing, is explored for the first time in VR. Optimized serialization modeling shows potential in capturing long-range dependencies among frames. On the other hand, we equip the sequence-to-sequence model with an unsupervised optical flow estimator to maximize its potential. The flow estimator is trained with our proposed unsupervised distillation loss, which can alleviate the data discrepancy and inaccurate degraded optical flow issues of previous flow-based methods. With reliable optical flow, we can establish accurate correspondence among multiple frames, narrowing the domain difference between 1D language and 2D misaligned frames and improving the potential of the sequence-to-sequence model. S2SVR shows superior performance in multiple VR tasks, including video deblurring, video super-resolution, and compressed video quality enhancement. Code and models are publicly available at https://github.com/linjing7/VR-Baseline
翻译:如何在视频序列中适当建模跨框架关系,这是对视频恢复的一个重要但尚未解决的挑战。 在这项工作中,我们提出一个不受监督的流程对齐序列至序列模型(S2SVR)来解决这个问题。一方面,在视频序列中首次探索了在视频序列中如何适当建模跨框架关系模型,该模型已证明能够在自然语言处理领域进行序列建模。优化的序列化模型显示在捕捉各框架之间远程依赖性方面的潜力。另一方面,我们为序列到序列模型配备了一个不受监督的光源流测算器,以最大限度地扩大其潜力。流程测算器经过培训后,我们提出了未经监督的蒸馏损失,这可以缓解数据差异和以往流程处理方法中不准确退化的光流问题。在可靠的光学流中,我们可以在多个框架之间建立准确的对应关系,缩小1D语言和2D之间的域差异,并改进序列到序列序列序列序列流模型的潜力。S2S2S-S-S-BRES-S-SBS-S-SBLVB Servical de Vlab Servical de Vlibal Sheliblegal Shemaism Regal Supligal Shemavial Shelibs de Vlibal Shemal Shemal Shemal Shemal Shemal Shemal Shemal Shemal Shemlibismal Shemal Shemal Shemal Shemismismmmmalismmmmmmmmmmmmmmmmismismismmmismal Sheldal Sheldal Shemal commal commdal commal commal commal commal commal commal commal commal commal commal commal commal commal commal commal commal commal commal commmmmmmmmal commal commal commmmmmmmmmal smal smal commal commal smal commmmmal smal