When facing changing environments in the real world, the lightweight model on client devices suffers from severe performance drops under distribution shifts. The main limitations of the existing device model lie in (1) unable to update due to the computation limit of the device, (2) the limited generalization ability of the lightweight model. Meanwhile, recent large models have shown strong generalization capability on the cloud while they can not be deployed on client devices due to poor computation constraints. To enable the device model to deal with changing environments, we propose a new learning paradigm of Cloud-Device Collaborative Continual Adaptation, which encourages collaboration between cloud and device and improves the generalization of the device model. Based on this paradigm, we further propose an Uncertainty-based Visual Prompt Adapted (U-VPA) teacher-student model to transfer the generalization capability of the large model on the cloud to the device model. Specifically, we first design the Uncertainty Guided Sampling (UGS) to screen out challenging data continuously and transmit the most out-of-distribution samples from the device to the cloud. Then we propose a Visual Prompt Learning Strategy with Uncertainty guided updating (VPLU) to specifically deal with the selected samples with more distribution shifts. We transmit the visual prompts to the device and concatenate them with the incoming data to pull the device testing distribution closer to the cloud training distribution. We conduct extensive experiments on two object detection datasets with continually changing environments. Our proposed U-VPA teacher-student framework outperforms previous state-of-the-art test time adaptation and device-cloud collaboration methods. The code and datasets will be released.
翻译:当面对现实世界不断变化的环境时,客户装置的轻量模型受到分布变换过程中性能严重下降的影响。现有设备模型的主要局限在于:(1) 由于设备计算限制,无法更新设备模型的主要限制在于:(1) 由于设备计算限制,无法更新设备,(2) 轻量模型的普及能力有限。与此同时,最近大型模型显示云层的高度概括性能力,而由于计算限制,无法在客户设备上部署这些能力。为了使设备模型能够应对不断变化的环境,我们提议了一个UDLD-Devic 合作持续适应的新学习模式,鼓励云和装置之间的协作,并改进设备模型的概括化。基于这一模式,我们进一步提议一个基于不确定性的视觉提示调整(U-VPA) 教师测试模型,将云层大模型的普及性能力转移到设备模型。具体地说,我们首先设计“UGS”系统模型,以筛选挑战性的数据,并将最差的样本从设备传输到云层。然后,我们提议一个视觉快速的学习战略,以不确定以不稳度的值为基础,用更精确的测试方式, 将我们所选的分布式数据转换为更精确的测试。我们所选的模型,我们用更精确的测试,我们所选的测试, 将数据转换到更精确的频率的模型,我们所选的测试,然后用更精确的测试,我们所选的频率的频率的频率的计算。