This paper focuses on Super-resolution for online video streaming data. Applying existing super-resolution methods to video streaming data is non-trivial for two reasons. First, to support application with constant interactions, video streaming has a high requirement for latency that most existing methods are less applicable, especially on low-end devices. Second, existing video streaming protocols (e.g., WebRTC) dynamically adapt the video quality to the network condition, thus video streaming in the wild varies greatly under different network bandwidths, which leads to diverse and dynamic degradations. To tackle the above two challenges, we proposed a novel video super-resolution method for online video streaming. First, we incorporate Look-Up Table (LUT) to lightweight convolution modules to achieve real-time latency. Second, for variant degradations, we propose a pixel-level LUT fusion strategy, where a set of LUT bases are built upon state-of-the-art SR networks pre-trained on different degraded data, and those LUT bases are combined with extracted weights from lightweight convolution modules to adaptively handle dynamic degradations. Extensive experiments are conducted on a newly proposed online video streaming dataset named LDV-WebRTC. All the results show that our method significantly outperforms existing LUT-based methods and offers competitive SR performance with faster speed compared to efficient CNN-based methods. Accelerated with our parallel LUT inference, our proposed method can even support online 720P video SR around 100 FPS.
翻译:本文侧重于用于在线视频流数据的超分辨率数据。 将现有超分辨率方法应用于视频流数据不具有三重性, 原因有二。 首先, 为了支持持续互动的应用,视频流对潜伏要求很高, 大部分现有方法都不太适用, 特别是在低端设备上。 第二, 现有的视频流协议( 例如WebRTC) 动态地将视频流质量适应网络条件, 因此在不同的网络带宽下, 野外视频流差异很大, 导致不同和动态的退化。 为了应对上述两个挑战, 我们提出了一个新的视频流流应用超分辨率方法。 首先, 我们把 LUp- Up 模块(LUT) 纳入轻量级演算组合模块, 以实现实时延迟。 第二, 为了变形, 我们提议了一个像素级 LUT 熔化战略, 这套LUT基础建于基于最先进的SR 网络网络, 并且这些 LUT基础与从较轻级的同级的视频支持模块中提取的重量, 将我们最新的视频流压式的SLLLV 测试方法与我们目前快速的视频流压式数据。</s>