In this paper, we consider the task of space-time video super-resolution (ST-VSR), which can increase the spatial resolution and frame rate for a given video simultaneously. Despite the remarkable progress of recent methods, most of them still suffer from high computational costs and inefficient long-range information usage. To alleviate these problems, we propose a Bidirectional Recurrence Network (BRN) with the optical-flow-reuse strategy to better use temporal knowledge from long-range neighboring frames for high-efficiency reconstruction. Specifically, an efficient and memory-saving multi-frame motion utilization strategy is proposed by reusing the intermediate flow of adjacent frames, which considerably reduces the computation burden of frame alignment compared with traditional LSTM-based designs. In addition, the proposed hidden state in BRN is updated by the reused optical flow and refined by the Feature Refinement Module (FRM) for further optimization. Moreover, by utilizing intermediate flow estimation, the proposed method can inference non-linear motion and restore details better. Extensive experiments demonstrate that our optical-flow-reuse-based bidirectional recurrent network (OFR-BRN) is superior to state-of-the-art methods in accuracy and efficiency.
翻译:在本文中,我们考虑了空间时间视频超分辨率(ST-VSR)的任务,它可以同时提高某一视频的空间分辨率和框架率。尽管最近的方法取得了显著的进展,但大多数方法仍然受到高计算成本和低效率远程信息使用的影响。为了缓解这些问题,我们提议采用光-流-再利用战略双向重复网络(BRN),以更好地利用长距离相邻框架的时间知识,促进高效重建。具体地说,通过重新使用相邻框架的中间流,提出了高效和记忆节省的多框架动作利用战略,大大降低了与传统的LSTM设计相比,框架调整的计算负担。此外,BRN的拟议隐藏状态通过再利用光学流动加以更新,并由FARM(FRM)改进,以进一步优化。此外,通过使用中间流量估计,拟议的方法可以更好地推断非线运动和恢复细节。广泛的实验表明,我们的光-流-流-流-源-双向-周期网络(OFR-BRN-Rart)的精确度方法高于状态。