This paper considers a new problem of adapting a pre-trained model of human mesh reconstruction to out-of-domain streaming videos. However, most previous methods based on the parametric SMPL model \cite{loper2015smpl} underperform in new domains with unexpected, domain-specific attributes, such as camera parameters, lengths of bones, backgrounds, and occlusions. Our general idea is to dynamically fine-tune the source model on test video streams with additional temporal constraints, such that it can mitigate the domain gaps without over-fitting the 2D information of individual test frames. A subsequent challenge is how to avoid conflicts between the 2D and temporal constraints. We propose to tackle this problem using a new training algorithm named Bilevel Online Adaptation (BOA), which divides the optimization process of overall multi-objective into two steps of weight probe and weight update in a training iteration. We demonstrate that BOA leads to state-of-the-art results on two human mesh reconstruction benchmarks.
翻译:本文审议了将人类网格重建的预培训模式改造为外向流视频的新问题。 然而,基于参数SMPL模型的多数以往方法基于参数 SMPL 模型 \ cite{loper2015smpl} 在有意外的、特定域属性(如相机参数、骨骼长度、背景和隔离等)的新领域表现不佳。 我们的一般想法是动态地微调测试视频流的源模型,同时增加时间限制,这样它就可以在不过分适应单个测试框架的2D信息的情况下缩小域间差距。 接下来的挑战是如何避免2D和时间限制之间的冲突。 我们提议使用名为双级在线适应(BOA)的新培训算法来解决这一问题,该算出将总体多目标优化进程分为两步的重量探测和在培训中更新重量。 我们证明BOA导致两个人类网格重建基准的状态。